Nov 03 00:38:10.417357 ip-10-0-136-112 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 00:38:10.417367 ip-10-0-136-112 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 00:38:10.417374 ip-10-0-136-112 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 00:38:10.417590 ip-10-0-136-112 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 00:38:20.551438 ip-10-0-136-112 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 00:38:20.551451 ip-10-0-136-112 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ddb5c5921a7d44139065de3e8d70892d --
Nov 03 00:40:13.384493 ip-10-0-136-112 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:40:13.835181 ip-10-0-136-112 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:13.835181 ip-10-0-136-112 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:40:13.835181 ip-10-0-136-112 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:13.835181 ip-10-0-136-112 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:13.839675 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839590    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.839675 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839675    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.839675 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839679    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839682    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839685    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839688    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839691    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839695    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839697    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839700    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839703    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839709    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839712    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839715    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839718    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839721    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839724    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839726    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839729    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839731    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839734    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839737    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.839776 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839739    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839742    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839745    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839747    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839750    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839753    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839755    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839758    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839761    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839763    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839766    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839769    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839771    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839774    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839778    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839781    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839784    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839787    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839789    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839792    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.840248 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839795    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839797    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839801    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839803    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839808    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839813    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839816    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839819    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839822    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839825    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839827    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839830    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839832    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839835    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839837    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839840    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839843    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839846    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839848    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.840800 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839851    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839854    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839858    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839861    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839864    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839866    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839870    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839872    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839875    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839878    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839881    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839885    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839889    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839892    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839895    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839898    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839901    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839904    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839906    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.841275 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839909    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839911    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839914    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839916    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839919    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839921    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839924    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839927    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839929    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839932    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839934    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839937    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839939    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839942    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839945    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839948    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839951    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839954    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839957    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839959    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.841822 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839963    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839966    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839970    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839972    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839975    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.839978    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840777    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840784    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840787    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840790    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840793    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840796    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840799    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840802    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840804    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840808    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840810    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840813    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840816    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.842649 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840819    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840822    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840824    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840827    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840829    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840832    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840835    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840838    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840841    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840844    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840847    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840849    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840852    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840854    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840857    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840860    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840862    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840865    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840868    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840870    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.843222 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840873    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840876    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840878    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840881    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840884    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840886    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840889    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840892    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840894    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840897    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840899    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840902    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840905    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840907    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840911    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840914    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840917    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840919    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840922    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840925    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.843738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840928    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840930    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840933    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840937    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840940    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840943    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840946    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840948    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840951    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840953    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840956    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840959    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840961    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840964    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840967    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840969    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840972    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840975    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840977    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840980    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.844217 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840982    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840985    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840987    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840990    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840992    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840995    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.840998    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841000    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841003    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841005    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841015    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841019    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841023    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841025    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841028    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841031    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841033    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841036    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841038    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841041    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.844685 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841044    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841046    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841049    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841051    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841054    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841057    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841059    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841062    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841064    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841067    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841073    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841078    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.841082    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842392    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842404    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842409    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842414    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842418    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.845224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842423    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842427    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842431    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842442    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842447    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842452    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842456    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842460    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842464    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842468    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842473    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842477    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842481    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842486    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842491    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842503    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842509    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842514    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.845650 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842518    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842522    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842527    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842531    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842535    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842539    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842544    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842549    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842562    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842566    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842570    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842574    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842578    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842583    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842590    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842594    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842598    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842603    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842607    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.846079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842612    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842620    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842625    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842629    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842633    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842637    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842641    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842645    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842649    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842654    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842658    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842663    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842667    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842672    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842680    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842685    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842689    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842694    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842698    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842703    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.846552 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842708    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842712    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842716    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842720    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842724    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842728    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842740    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842745    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842749    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842753    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842760    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842764    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842768    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842773    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842777    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842782    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842803    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.842998    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843004    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843008    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.847038 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843011    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843015    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843018    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843021    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843023    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843026    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843029    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843032    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843035    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843037    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843044    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843047    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843071    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843074    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843077    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843079    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843082    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843085    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843088    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843091    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.847519 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843093    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843096    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.843099    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844222    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844232    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844240    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844245    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844249    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844253    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844262    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844266    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844269    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844272    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844276    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844280    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844283    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844286    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844289    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844293    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844296    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844299    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844304    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844308    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844311    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:40:13.847982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844314    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844317    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844326    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844329    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844333    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844336    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844339    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844342    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844345    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844348    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844351    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844356    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844359    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844362    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844365    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844369    2578 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844372    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844378    2578 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844382    2578 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844385    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844389    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844392    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844397    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844400    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844403    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:40:13.848540 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844406    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844409    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844412    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844415    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844418    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844421    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844424    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844427    2578 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844431    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844434    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844437    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844440    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844444    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844447    2578 flags.go:64] FLAG: --help="false"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844450    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844453    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844456    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844459    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844462    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844466    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844469    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844472    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844474    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:40:13.849152 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844477    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844481    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844484    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844488    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844491    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844494    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844497    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844500    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844503    2578 flags.go:64] FLAG: --lock-file=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844506    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844509    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844512    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844518    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844521    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844524    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844527    2578 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844531    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844534    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844537    2578 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844540    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844544    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844547    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844551    2578 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844555    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844558    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:40:13.849741 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844561    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844564    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844567    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844570    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844573    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844581    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844584    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844587    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844590    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844593    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844597    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844600    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844603    2578 flags.go:64] FLAG: --port="10250"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844606    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844609    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03440cf63280987ad"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844612    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844615    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844619    2578 flags.go:64] FLAG: --register-node="true"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844622    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844626    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844628    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844631    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844634    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844638    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844641    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844645    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:40:13.850320 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844648    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844651    2578 flags.go:64] FLAG: --runonce="false"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844654    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844657    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844660    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844663    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844666    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844673    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844676    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844680    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844683    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844685    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844688    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844691    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844695    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844698    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844701    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844708    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844711    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844714    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844718    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844721    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844724    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844727    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844731    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:40:13.850936 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844734    2578 flags.go:64] FLAG: --v="2"
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844739    2578 flags.go:64] FLAG: --version="false"
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844743    2578 flags.go:64] FLAG: --vmodule=""
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844747    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.844751    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844902    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844906    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844909    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844912    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844915    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844920    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844923    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844926    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844929    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844934    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844936    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844939    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844942    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844945    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844947    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844950    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.851586 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844953    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844956    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844958    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844961    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844964    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844966    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844969    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844972    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844975    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844978    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844980    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844983    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844985    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844988    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844990    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844993    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844996    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.844998    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845001    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845003    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.852125 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845006    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845016    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845019    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845022    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845025    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845028    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845031    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845034    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845036    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845039    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845041    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845044    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845046    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845049    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845051    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845054    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845057    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845059    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845062    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845065    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.852608 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845067    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845070    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845074    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845078    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845082    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845085    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845087    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845090    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845092    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845095    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845098    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845100    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845103    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845106    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845108    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845111    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845113    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845117    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845120    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.853089 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845122    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845125    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845128    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845130    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845133    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845135    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845138    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845141    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845143    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845146    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845148    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845152    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845155    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845172    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845175    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845177    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845180    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845183    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845186    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.853599 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845188    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845191    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845194    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845196    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845199    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845201    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845204    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845206    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845209    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845211    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845214    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845218    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.845247    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845288    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845293    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845297    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.854079 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845300    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845303    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845306    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845308    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845311    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845314    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845316    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845320    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845324    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845327    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845330    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845332    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845335    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845337    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845341    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845343    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845346    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845349    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845351    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.854462 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845354    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845356    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845359    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845362    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845366    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845369    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845372    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845375    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845378    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845381    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845384    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845386    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845389    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845392    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845394    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845397    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845399    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845402    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.854916 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845405    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845407    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845411    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845414    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845417    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845419    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845422    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845424    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845427    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845430    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845433    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845436    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845440    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845444    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845447    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845450    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845453    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845455    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845458    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845460    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.855373 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845463    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845465    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845469    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845472    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845475    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845477    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845480    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845482    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845485    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845487    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845490    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845492    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845495    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845497    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845501    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845504    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845506    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845509    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845511    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845514    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.855834 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845516    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845519    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845522    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845524    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845527    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845530    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845532    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845535    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845537    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845540    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845543    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845545    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845547    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845550    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845554    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845556    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845559    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845561    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845564    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845566    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.856307 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845569    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845572    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845575    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845578    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.845580    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.852805    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.852823    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.852839    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.852844    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852901    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852906    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852910    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852913    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852916    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852918    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852921    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852924    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852927    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852929    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.856774 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852932    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852936    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852940    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852944    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852948    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852951    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852954    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852957    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852960    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852963    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852965    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852968    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852970    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852973    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852976    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852978    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852981    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852984    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852986    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.857237 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852991    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852994    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852996    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.852999    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853002    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853005    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853007    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853018    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853021    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853024    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853027    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853030    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853032    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853035    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853038    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853041    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853043    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853046    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853049    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.857738 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853051    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853056    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853063    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853065    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853069    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853071    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853074    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853077    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853079    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853082    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853084    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853087    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853090    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853094    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853097    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853100    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853103    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853105    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.858268 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853108    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853110    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853113    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853116    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853119    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853121    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853124    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853127    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853129    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853132    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853134    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853137    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853140    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853142    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853145    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853148    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853151    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853153    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853156    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.858766 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853172    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853174    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853177    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853180    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853182    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853185    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853188    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853191    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853193    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853197    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853199    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853202    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853205    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853208    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853210    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853213    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853216    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853219    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853221    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.859329 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853224    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.853252    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853388    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853393    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853397    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853401    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853404    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853406    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853409    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853412    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853415    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853418    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853421    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853423    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853426    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853428    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.859809 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853431    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853434    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853436    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853439    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853441    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853444    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853447    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853451    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853453    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853456    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853458    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853461    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853463    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853466    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853468    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853471    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853473    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853476    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853479    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853483    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.860224 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853486    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853489    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853491    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853494    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853497    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853499    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853502    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853505    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853507    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853510    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853512    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853515    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853518    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853520    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853522    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853525    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853528    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853530    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853533    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853535    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.860758 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853539    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853541    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853545    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853549    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853552    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853555    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853557    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853560    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853562    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853565    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853567    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853570    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853572    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853575    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853577    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853580    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853583    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853586    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853588    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.861314 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853590    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853593    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853595    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853598    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853601    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853603    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853606    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853609    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853611    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853614    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853617    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853619    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853622    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853624    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853627    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853630    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853633    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853635    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853637    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.862014 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853640    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853642    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853645    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853647    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853650    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853652    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853655    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853657    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853660    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853662    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853665    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853668    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853671    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853673    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.853696    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853727    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:13.862699 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853731    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853734    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853737    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853740    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853743    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853746    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853748    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853750    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853753    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853756    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853758    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853761    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853764    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853767    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853770    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853772    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853774    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853777    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853779    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853782    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:13.863082 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853785    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853787    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853806    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853810    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853813    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853817    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853820    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853822    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853825    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853827    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853830    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853832    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853835    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853839    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853842    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853845    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853847    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853850    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853852    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853855    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:13.863837 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853858    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853860    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853863    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853865    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853869    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853873    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853875    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853878    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853880    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853883    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853886    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853888    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853891    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853893    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853896    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853898    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853901    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853905    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853908    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:13.864400 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853911    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853914    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853916    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853919    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853922    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853924    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853927    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853929    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853932    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853934    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853936    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853939    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853942    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853944    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853947    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853949    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853952    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853954    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853957    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:13.864838 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853960    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853963    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853965    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853968    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853971    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853973    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853976    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853978    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853980    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853983    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853985    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853988    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853990    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853992    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853995    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.853997    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854000    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854002    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854005    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:13.865317 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854011    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854013    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854016    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854019    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854021    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854024    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854026    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:13.854029    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.854087    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.857534    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.858613    2578 server.go:1014] "Starting client certificate rotation"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.858725    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.858772    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.860904    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:40:13.865752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.863728    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:40:13.876260 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.876238    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:40:13.880267 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.880252    2578 log.go:25] "Validated CRI v1 image API"
Nov 03 00:40:13.881418 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.881361    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:40:13.889129 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.889107    2578 fs.go:135] Filesystem UUIDs: map[1bb74329-46d0-487e-ba8a-1297fbf39683:/dev/nvme0n1p4 36e90348-92e9-40ca-83ca-03c399f27cd3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 00:40:13.889226 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.889127    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 00:40:13.890244 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.890225    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:40:13.894303 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.894194    2578 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:13.892737024 +0000 UTC m=+0.381363465 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099376 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c9ed336111058ad4b8b38f43687d SystemUUID:ec28c9ed-3361-1105-8ad4-b8b38f43687d BootID:ddb5c592-1a7d-4413-9065-de3e8d70892d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:d5:ed:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:d5:ed:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:03:51:55:dc:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:40:13.894303 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.894302    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:40:13.894432 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.894378    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:40:13.897287 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897267    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:40:13.897450 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897289    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-112.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:40:13.897496 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897453    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:40:13.897496 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897461    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:40:13.897549 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897496    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:40:13.897549 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.897513    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:40:13.898769 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.898760    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:40:13.903501 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903490    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:40:13.903625 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903615    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:40:13.903658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903649    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:40:13.903685 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903663    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:40:13.903685 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903675    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:40:13.903685 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903684    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:40:13.903801 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903692    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:40:13.904003 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.903991    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:40:13.906571 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.906553    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:40:13.912441 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.912418    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-njvlg"
Nov 03 00:40:13.913800 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.913788    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:40:13.914745 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914730    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:40:13.914799 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914764    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:40:13.914937 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914922    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:40:13.914982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914948    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:40:13.914982 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914962    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:40:13.915054 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.914992    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:40:13.915054 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915006    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:40:13.915054 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915015    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:40:13.915716 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915698    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:40:13.915795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915719    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:40:13.915795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915737    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:40:13.915795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915750    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:40:13.915795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915777    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:40:13.915795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.915791    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:40:13.916586 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.916574    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:40:13.916635 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.916591    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:40:13.919074 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.919056    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-njvlg"
Nov 03 00:40:13.920678 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.920667    2578 server.go:1263] "Started kubelet"
Nov 03 00:40:13.920793 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.920761    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:40:13.920836 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.920774    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:40:13.920866 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.920844    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:40:13.921529 ip-10-0-136-112 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:40:13.922993 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.922972    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:40:13.924199 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.923772    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:40:13.927627 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.927588    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-112.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 00:40:13.927737 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.927713    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 00:40:13.927841 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.927821    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-112.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 00:40:13.929183 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.929135    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 00:40:13.931752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.931735    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:40:13.932352 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.932334    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:40:13.932483 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.932464    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:13.933058 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933043    2578 factory.go:55] Registering systemd factory
Nov 03 00:40:13.933124 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933061    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:40:13.933206 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933192    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:40:13.933249 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933194    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:40:13.933249 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933219    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:40:13.933324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933285    2578 factory.go:153] Registering CRI-O factory
Nov 03 00:40:13.933324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933298    2578 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:40:13.933324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933312    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:40:13.933324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933321    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:40:13.933502 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933345    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:40:13.933502 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933361    2578 factory.go:103] Registering Raw factory
Nov 03 00:40:13.933502 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.933373    2578 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:40:13.934537 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.934516    2578 manager.go:319] Starting recovery of all containers
Nov 03 00:40:13.937801 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.937694    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:13.940758 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.940739    2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-112.us-east-2.compute.internal\" not found" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:13.944515 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.944499    2578 manager.go:324] Recovery completed
Nov 03 00:40:13.949286 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.949274    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:13.950959 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.950945    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:13.951029 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.950982    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:13.951029 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.950994    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:13.951609 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.951593    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:40:13.951609 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.951607    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:40:13.951733 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.951613    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:13.951733 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.951627    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:40:13.952354 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.952341    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:13.952432 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.952369    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:13.952432 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.952383    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:13.953819 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.953807    2578 policy_none.go:50] "Start"
Nov 03 00:40:13.953868 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.953821    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:40:13.953868 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.953831    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:40:13.955855 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.955845    2578 policy_none.go:44] "Start"
Nov 03 00:40:13.993716 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.993687    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:40:13.997423 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997408    2578 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:40:13.997504 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.997456    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:40:13.997504 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997467    2578 server.go:87] "Starting device plugin registration server"
Nov 03 00:40:13.997731 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997717    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:40:13.997783 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997733    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:40:13.997888 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997872    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:40:13.997995 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997967    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:40:13.997995 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:13.997983    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:40:13.998906 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.998881    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:40:13.999012 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:13.998948    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.096781 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.096692    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:40:14.098853 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.097983    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:14.098853 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.098004    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:40:14.098853 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.098020    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:40:14.098853 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.098045    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:40:14.098853 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.098086    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 00:40:14.099083 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.098987    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:14.099083 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.099022    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:14.099083 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.099040    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:14.099083 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.099072    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.104231 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.104215    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:14.108460 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.108447    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.108503 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.108468    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-112.us-east-2.compute.internal\": node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.121411 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.121270    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.198522 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.198484    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"]
Nov 03 00:40:14.198646 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.198558    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:14.199565 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.199549    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:14.199657 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.199576    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:14.199657 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.199590    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:14.201920 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.201907    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:14.202075 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.202138 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202096    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:14.202689 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202674    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:14.202752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202701    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:14.202752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202712    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:14.202752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202734    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:14.202752 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202753    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:14.202892 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.202765    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:14.204827 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.204813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.204888 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.204838    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:14.206438 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.206423    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:14.206503 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.206449    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:14.206503 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.206461    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:14.221607 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.221583    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.223661 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.223638    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.229793 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.229774    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.322067 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.322037    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.334869 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.334845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.334976 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.334875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.334976 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.334910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.423129 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.423058    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.435400 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.435476 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.435476 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.435538 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.435538 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.435538 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.435485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.523557 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.523529    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.526719 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.526704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.532301 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.532285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:14.624071 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.624038    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.724653 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.724582    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.829895 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.829855    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:14.859401 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.859366    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 00:40:14.859854 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.859516    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:40:14.859854 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.859555    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:40:14.928372 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.928297    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:14 +0000 UTC" deadline="2028-05-11 10:17:46.043865427 +0000 UTC"
Nov 03 00:40:14.928372 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:14.928361    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13329h37m31.115507724s"
Nov 03 00:40:14.930059 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:14.930042    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:15.008014 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.007943    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:15.031230 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.031196    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:15.131874 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.131845    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:15.180344 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:15.180303    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc7fd51fa7784f0272b9c9a23f1eea8.slice/crio-f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77 WatchSource:0}: Error finding container f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77: Status 404 returned error can't find the container with id f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77
Nov 03 00:40:15.187646 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.187626    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:40:15.193378 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:15.193355    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecdbd9fc26c68793bebc9a7943385b7.slice/crio-4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3 WatchSource:0}: Error finding container 4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3: Status 404 returned error can't find the container with id 4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3
Nov 03 00:40:15.232950 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.232921    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:15.261240 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.261160    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:15.334531 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.334495    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:15.344344 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.344324    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:15.345068 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.345055    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:15.357068 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.357054    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:15.905066 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.905036    2578 apiserver.go:52] "Watching apiserver"
Nov 03 00:40:15.915113 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.915080    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:40:15.915883 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.915491    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs","openshift-image-registry/node-ca-l9wpw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal","openshift-multus/multus-5j8gb","openshift-multus/network-metrics-daemon-rlbmn","openshift-network-diagnostics/network-check-target-9g7gh","kube-system/konnectivity-agent-cbkz7","openshift-network-operator/iptables-alerter-m9fbf","kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-7fbfg","kube-system/global-pull-secret-syncer-7vg7f","openshift-dns/node-resolver-wrzvg","openshift-multus/multus-additional-cni-plugins-cw99z","openshift-ovn-kubernetes/ovnkube-node-5ff5v"]
Nov 03 00:40:15.919756 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.918199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:15.922100 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.921021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.926130 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.926108    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:40:15.926391 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.926374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.926703 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.926200    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:40:15.927442 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.927416    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:40:15.927599 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.927583    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:40:15.927702 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.927680    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:40:15.927814 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.927787    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:40:15.928174 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.928157    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:40:15.934375 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.928343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.934477 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.929108    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\""
Nov 03 00:40:15.934700 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.929482    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\""
Nov 03 00:40:15.934791 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.929774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:15.934989 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.933185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:15.935066 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.934995    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:15.935066 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.933672    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\""
Nov 03 00:40:15.935066 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.933731    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:40:15.935282 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.933891    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:40:15.935282 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.935062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:15.935282 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.934185    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:40:15.935282 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.934207    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:40:15.935282 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.933253    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 00:40:15.937793 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.937773    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:40:15.938353 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.938333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:15.941706 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.941687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:15.944947 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.944927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:15.946899 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.946880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:15.946997 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.946897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.946997 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.946930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.946997 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.946955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpp4\" (UniqueName: \"kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.946997 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.946977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjp97\" (UniqueName: \"kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.947256 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5972d\" (UniqueName: \"kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9m6\" (UniqueName: \"kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:15.947658 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.947989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskw4\" (UniqueName: \"kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.948368 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:15.949183 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.948757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:15.949682 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.949581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:15.949682 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:15.949638    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:15.952458 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.952127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:15.973157 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.973133    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:15.984709 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.984686    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:40:15.984810 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:15.984774    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bgxps"
Nov 03 00:40:16.003128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.003102    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bgxps"
Nov 03 00:40:16.035260 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.034171    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:40:16.041349 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041325    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:40:16.041463 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041453    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:40:16.041528 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041482    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:40:16.041528 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041524    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:40:16.041641 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041580    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\""
Nov 03 00:40:16.041692 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041658    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:40:16.041743 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041711    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\""
Nov 03 00:40:16.041794 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041759    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:40:16.041794 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041766    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:16.041794 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041783    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041823    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041861    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041884    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041919    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041929    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:40:16.041952 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041942    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:40:16.042205 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041989    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:40:16.042205 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.041946    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:40:16.042205 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.042032    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\""
Nov 03 00:40:16.042333 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.042218    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\""
Nov 03 00:40:16.042547 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.042529    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\""
Nov 03 00:40:16.049520 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.049629 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049629 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049629 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049580    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.049629 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049659    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049670    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.049831 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049706    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049843    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049847    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.049995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpp4\" (UniqueName: \"kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050194 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjp97\" (UniqueName: \"kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn8d\" (UniqueName: \"kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xfn\" (UniqueName: \"kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050532    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050650    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050717    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.050967 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050832    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050927    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.050943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9m6\" (UniqueName: \"kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051000    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051006    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.051779 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051331    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskw4\" (UniqueName: \"kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.051690    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051719    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051761    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.051803    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.551783764 +0000 UTC m=+3.040410194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051850    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.052445 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.051993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052021    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052174    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052285    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.053319 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5972d\" (UniqueName: \"kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz8n\" (UniqueName: \"kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgcw\" (UniqueName: \"kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054128 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.052989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.053016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.053094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.053632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.053683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.053732    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054024    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054054    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054098    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054455    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.054924 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.054463    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.057270 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.056328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:16.057270 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.056707    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.066732 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.066329    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:16.066732 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.066352    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:16.066732 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.066366    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:16.066732 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.066439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.566420172 +0000 UTC m=+3.055046609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:16.068795 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.067889    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskw4\" (UniqueName: \"kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.068896 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.068852    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpp4\" (UniqueName: \"kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.070280 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.070235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5972d\" (UniqueName: \"kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.070934 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.070910    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjp97\" (UniqueName: \"kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.072100 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.072075    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9m6\" (UniqueName: \"kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:16.104099 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.104038    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4ecdbd9fc26c68793bebc9a7943385b7","Type":"ContainerStarted","Data":"4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3"}
Nov 03 00:40:16.105403 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.105366    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerStarted","Data":"f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77"}
Nov 03 00:40:16.153987 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.153953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.153996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn8d\" (UniqueName: \"kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xfn\" (UniqueName: \"kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154110    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154148 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.154506 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz8n\" (UniqueName: \"kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgcw\" (UniqueName: \"kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.154887 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154886    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.155215 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.154936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.155379 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.155526 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.155593 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155568    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.155716 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.155652    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:16.155716 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155681    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.155825 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.155707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:16.655687289 +0000 UTC m=+3.144313723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:16.155825 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155739    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.155963 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155871    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.155963 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.156073 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155972    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.156451 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.156556 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156540    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.156627 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.156827 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.156922 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.155925    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.156922 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.157024 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.157024 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.156994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.157149 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.157024    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.157149 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.157039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.157149 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.157118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.157483 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.157464    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.160557 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.160461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.160557 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.160511    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.170159 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.170139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn8d\" (UniqueName: \"kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.173155 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.173120    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz8n\" (UniqueName: \"kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.173923 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.173903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgcw\" (UniqueName: \"kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.175519 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.175497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xfn\" (UniqueName: \"kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.241201 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.241155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:16.253562 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.253530    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c45381d_5957_4d81_847f_5e3ad1b1780c.slice/crio-32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722 WatchSource:0}: Error finding container 32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722: Status 404 returned error can't find the container with id 32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722
Nov 03 00:40:16.276103 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.276076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:16.280775 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.280754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:16.283512 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.283463    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:16.287559 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.287529    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda548ec83_53fc_4a5c_a5d8_574b6ee5e971.slice/crio-85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355 WatchSource:0}: Error finding container 85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355: Status 404 returned error can't find the container with id 85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355
Nov 03 00:40:16.289303 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.289228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:16.293417 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.293301    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eec9d2b_a414_4463_9745_5a9dc01f4061.slice/crio-a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4 WatchSource:0}: Error finding container a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4: Status 404 returned error can't find the container with id a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4
Nov 03 00:40:16.299938 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.299906    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91635f66_bcd2_4510_ad4b_d0412ace3255.slice/crio-f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4 WatchSource:0}: Error finding container f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4: Status 404 returned error can't find the container with id f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4
Nov 03 00:40:16.300866 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.300847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:16.304575 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.304547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:16.306941 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.306922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:16.309600 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.309568    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:16.314124 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.314100    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8639c_a7fe_4b29_ba5a_8f865346ae9d.slice/crio-aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade WatchSource:0}: Error finding container aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade: Status 404 returned error can't find the container with id aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade
Nov 03 00:40:16.315054 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.315034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:16.320620 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.320495    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7044cbc_8ac5_4ada_a632_8011de51e2d9.slice/crio-9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf WatchSource:0}: Error finding container 9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf: Status 404 returned error can't find the container with id 9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf
Nov 03 00:40:16.328130 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.328042    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5448759_b9dd_422e_92dc_e87aedb5ef98.slice/crio-191013a02106c363195750638b1ef3810438f8eba583673b5ce2249183f7d711 WatchSource:0}: Error finding container 191013a02106c363195750638b1ef3810438f8eba583673b5ce2249183f7d711: Status 404 returned error can't find the container with id 191013a02106c363195750638b1ef3810438f8eba583673b5ce2249183f7d711
Nov 03 00:40:16.329339 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.329217    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1355bd78_3c21_4bf1_a585_8e66bf03cbb1.slice/crio-b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd WatchSource:0}: Error finding container b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd: Status 404 returned error can't find the container with id b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd
Nov 03 00:40:16.334102 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:16.334072    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfad136_a64a_4525_ae3a_d77fb5af1084.slice/crio-26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603 WatchSource:0}: Error finding container 26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603: Status 404 returned error can't find the container with id 26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603
Nov 03 00:40:16.557358 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.557267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:16.557540 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.557428    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:16.557540 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.557511    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:17.55749116 +0000 UTC m=+4.046117603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:16.658241 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.658199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:16.658427 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:16.658262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:16.658427 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658386    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:16.658544 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658428    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:16.658544 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658450    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:16.658544 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658465    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:16.658544 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:17.658452624 +0000 UTC m=+4.147079063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:16.658544 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:16.658521    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:17.65850485 +0000 UTC m=+4.147131285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:17.007645 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.007514    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:16 +0000 UTC" deadline="2028-04-12 06:03:15.75847011 +0000 UTC"
Nov 03 00:40:17.007645 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.007582    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12629h22m58.750892282s"
Nov 03 00:40:17.098704 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.098297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:17.098704 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.098409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:17.117617 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.117540    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9wpw" event={"ID":"1355bd78-3c21-4bf1-a585-8e66bf03cbb1","Type":"ContainerStarted","Data":"b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd"}
Nov 03 00:40:17.122418 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.122304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf"}
Nov 03 00:40:17.144004 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.143963    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs" event={"ID":"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d","Type":"ContainerStarted","Data":"aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade"}
Nov 03 00:40:17.154167 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.154134    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5j8gb" event={"ID":"7eec9d2b-a414-4463-9745-5a9dc01f4061","Type":"ContainerStarted","Data":"a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4"}
Nov 03 00:40:17.159432 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.159402    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerStarted","Data":"85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355"}
Nov 03 00:40:17.173141 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.173017    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wrzvg" event={"ID":"2bfad136-a64a-4525-ae3a-d77fb5af1084","Type":"ContainerStarted","Data":"26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603"}
Nov 03 00:40:17.174794 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.174762    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7fbfg" event={"ID":"c5448759-b9dd-422e-92dc-e87aedb5ef98","Type":"ContainerStarted","Data":"191013a02106c363195750638b1ef3810438f8eba583673b5ce2249183f7d711"}
Nov 03 00:40:17.176547 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.176523    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m9fbf" event={"ID":"91635f66-bcd2-4510-ad4b-d0412ace3255","Type":"ContainerStarted","Data":"f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4"}
Nov 03 00:40:17.178180 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.178145    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cbkz7" event={"ID":"3c45381d-5957-4d81-847f-5e3ad1b1780c","Type":"ContainerStarted","Data":"32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722"}
Nov 03 00:40:17.568202 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.567589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:17.568202 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.567750    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:17.568202 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.567818    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:19.567797886 +0000 UTC m=+6.056424345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.668539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:17.668597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.668823    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.668844    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.668866    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.668928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:19.668907885 +0000 UTC m=+6.157534322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.669377    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:17.669465 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:17.669430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:19.669414221 +0000 UTC m=+6.158040668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:18.008412 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.008330    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:16 +0000 UTC" deadline="2028-07-15 03:44:20.807669842 +0000 UTC"
Nov 03 00:40:18.008412 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.008376    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14883h4m2.799299962s"
Nov 03 00:40:18.101009 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.100928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:18.101169 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:18.101082    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:18.101684 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.101517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:18.101684 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:18.101621    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:18.194494 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.194445    2578 generic.go:356] "Generic (PLEG): container finished" podID="4bc7fd51fa7784f0272b9c9a23f1eea8" containerID="61910b81cdc69058c6e5b6b1a4ce964ebf43bbe0a941436fc5e001ab54b16c42" exitCode=0
Nov 03 00:40:18.195035 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.194730    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerDied","Data":"61910b81cdc69058c6e5b6b1a4ce964ebf43bbe0a941436fc5e001ab54b16c42"}
Nov 03 00:40:18.198043 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:18.198015    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4ecdbd9fc26c68793bebc9a7943385b7","Type":"ContainerStarted","Data":"26d7f07a20f2f45797335998eaa7188a21dbed31b44914427380b736097f0c0f"}
Nov 03 00:40:19.098928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.098893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:19.099437 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.099035    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:19.210384 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.210317    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerStarted","Data":"3b01ab80c9438f43e30341d35b3f71d6118062110380502a37805e698cf3a51d"}
Nov 03 00:40:19.226889 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.226822    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal" podStartSLOduration=4.226807948 podStartE2EDuration="4.226807948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:18.237954115 +0000 UTC m=+4.726580570" watchObservedRunningTime="2026-11-03 00:40:19.226807948 +0000 UTC m=+5.715434435"
Nov 03 00:40:19.589688 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.589057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:19.589688 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.589246    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:19.589688 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.589318    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.589297249 +0000 UTC m=+10.077923697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.690518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:19.690617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.690738    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.690801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.690782986 +0000 UTC m=+10.179409421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.691199    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.691218    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.691231    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:19.691333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:19.691297    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:23.691280569 +0000 UTC m=+10.179907020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:20.102121 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:20.101343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:20.102121 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:20.101490    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:20.102121 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:20.101942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:20.102121 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:20.102057    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:20.220449 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:20.219682    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="b33e6070a90738bb8cba6663447e7c3fca8be0cad9124cdb673b4b3e9a397858" exitCode=0
Nov 03 00:40:20.220449 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:20.219869    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"b33e6070a90738bb8cba6663447e7c3fca8be0cad9124cdb673b4b3e9a397858"}
Nov 03 00:40:20.246735 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:20.246458    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" podStartSLOduration=5.24643871 podStartE2EDuration="5.24643871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:19.227718271 +0000 UTC m=+5.716344724" watchObservedRunningTime="2026-11-03 00:40:20.24643871 +0000 UTC m=+6.735065165"
Nov 03 00:40:21.098866 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:21.098827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:21.099052 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:21.098973    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:21.231815 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:21.231770    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cbkz7" event={"ID":"3c45381d-5957-4d81-847f-5e3ad1b1780c","Type":"ContainerStarted","Data":"4f8af9a94d6fd6555e5ee4760901ca83940cba0ce1711f6e8512533562392820"}
Nov 03 00:40:22.099549 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:22.098991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:22.099549 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:22.099141    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:22.100044 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:22.099889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:22.100044 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:22.100003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:22.244822 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:22.244745    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l9wpw" event={"ID":"1355bd78-3c21-4bf1-a585-8e66bf03cbb1","Type":"ContainerStarted","Data":"d9ef8880ab47244eec3fe9cc4719c448c97ce9f84ba5b18ca1eddc7694c1febd"}
Nov 03 00:40:22.261757 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:22.260626    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cbkz7" podStartSLOduration=4.121024309 podStartE2EDuration="8.260607179s" totalImagesPullingTime="4.13958287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:21.251140188 +0000 UTC m=+7.739766643" watchObservedRunningTime="2026-11-03 00:40:22.260607179 +0000 UTC m=+8.749233636"
Nov 03 00:40:22.883848 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:22.883710    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:23.099593 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:23.099079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:23.099593 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.099227    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:23.248571 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:23.248484    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs" event={"ID":"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d","Type":"ContainerStarted","Data":"25f4b2d3afcee90e5be9cdc7ed2633ed4cdd5c92da0a67e620a7486edb83d6f7"}
Nov 03 00:40:23.625769 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:23.625243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:23.625769 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.625425    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:23.625769 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.625497    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:31.625478437 +0000 UTC m=+18.114104880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:23.726375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:23.726407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726508    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726520    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726529    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726568    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:31.726556354 +0000 UTC m=+18.215182794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726611    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:23.726896 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:23.726631    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:31.726624917 +0000 UTC m=+18.215251346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:24.099307 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:24.099271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:24.099479 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:24.099428    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:24.101125 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:24.101102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:24.101269 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:24.101217    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:24.478291 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:24.478235    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:40:25.008229 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.005598    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:24.478254958Z","UUID":"1147b3c2-b373-4ea1-ac91-3b4de0aec2a0","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:40:25.008380 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.007283    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:40:25.008380 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.008274    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:40:25.098841 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.098348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:25.098841 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:25.098476    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:25.255124 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.255047    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs" event={"ID":"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d","Type":"ContainerStarted","Data":"5b790b9afeb701d58b52926e54cfa2e5458bc5f6ee5d4000ae58be96858e0168"}
Nov 03 00:40:25.256711 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.256683    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5j8gb" event={"ID":"7eec9d2b-a414-4463-9745-5a9dc01f4061","Type":"ContainerStarted","Data":"d6e397b214b1ef1b2706a2d8cb58fc206b27e5e06900c8b0de481e2e92d61d93"}
Nov 03 00:40:25.259067 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.259041    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="52be3ea438888548f08c53c62674ca67513bb18b730b8af11159f0d0e2d03487" exitCode=0
Nov 03 00:40:25.259199 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.259126    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"52be3ea438888548f08c53c62674ca67513bb18b730b8af11159f0d0e2d03487"}
Nov 03 00:40:25.261390 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.260968    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wrzvg" event={"ID":"2bfad136-a64a-4525-ae3a-d77fb5af1084","Type":"ContainerStarted","Data":"976fb703a3a14accc62cec199cacdd245b53f07044be01812b00c8361bf82e00"}
Nov 03 00:40:25.273760 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.273704    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l9wpw" podStartSLOduration=6.076871044 podStartE2EDuration="11.273691621s" totalImagesPullingTime="5.196820577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:22.261715854 +0000 UTC m=+8.750342308" watchObservedRunningTime="2026-11-03 00:40:25.273691621 +0000 UTC m=+11.762318071"
Nov 03 00:40:25.273884 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.273833    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5j8gb" podStartSLOduration=3.295550111 podStartE2EDuration="11.273828085s" totalImagesPullingTime="7.978277974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:25.273106426 +0000 UTC m=+11.761732876" watchObservedRunningTime="2026-11-03 00:40:25.273828085 +0000 UTC m=+11.762454534"
Nov 03 00:40:25.309926 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.309857    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wrzvg" podStartSLOduration=2.94726627 podStartE2EDuration="11.30984015s" totalImagesPullingTime="8.36257388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:25.309304525 +0000 UTC m=+11.797930976" watchObservedRunningTime="2026-11-03 00:40:25.30984015 +0000 UTC m=+11.798466603"
Nov 03 00:40:25.846485 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.846179    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:25.847852 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:25.847820    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:26.099409 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.098999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:26.099409 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:26.099115    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:26.099569 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.099503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:26.099703 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:26.099593    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:26.266552 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.266502    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs" event={"ID":"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d","Type":"ContainerStarted","Data":"2b1349b714fda96abddfcaa4cbc9f3f2e1c72f44a9ec70c1f2d523b0c823c504"}
Nov 03 00:40:26.269329 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.269202    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="54280ec88b5cb097eb3a75eb2b47e4607a9df72a5918f69488d0feea05a5fc7c" exitCode=0
Nov 03 00:40:26.269329 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.269304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"54280ec88b5cb097eb3a75eb2b47e4607a9df72a5918f69488d0feea05a5fc7c"}
Nov 03 00:40:26.269968 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.269949    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:26.286740 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:26.285935    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs" podStartSLOduration=3.558220994 podStartE2EDuration="12.285903548s" totalImagesPullingTime="8.727682554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:26.285223683 +0000 UTC m=+12.773850134" watchObservedRunningTime="2026-11-03 00:40:26.285903548 +0000 UTC m=+12.774529999"
Nov 03 00:40:27.099736 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:27.099112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:27.099736 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:27.099240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:27.273517 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:27.273480    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7fbfg" event={"ID":"c5448759-b9dd-422e-92dc-e87aedb5ef98","Type":"ContainerStarted","Data":"16e96ea87118ad9b82aad2d450ab190638d1e6cbce720b409930ef5abea2bd84"}
Nov 03 00:40:27.275139 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:27.275111    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m9fbf" event={"ID":"91635f66-bcd2-4510-ad4b-d0412ace3255","Type":"ContainerStarted","Data":"ff99253c13742ec4797857b10913081661ecb810946196f40ec7dbe331015688"}
Nov 03 00:40:27.307203 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:27.307142    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m9fbf" podStartSLOduration=4.777794147 podStartE2EDuration="13.307128577s" totalImagesPullingTime="8.52933443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:27.306827842 +0000 UTC m=+13.795454354" watchObservedRunningTime="2026-11-03 00:40:27.307128577 +0000 UTC m=+13.795755028"
Nov 03 00:40:27.307341 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:27.307266    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7fbfg" podStartSLOduration=2.447862965 podStartE2EDuration="13.307257874s" totalImagesPullingTime="10.859394909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:27.29126235 +0000 UTC m=+13.779888801" watchObservedRunningTime="2026-11-03 00:40:27.307257874 +0000 UTC m=+13.795884325"
Nov 03 00:40:28.099324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:28.099286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:28.099324 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:28.099322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:28.099539 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:28.099453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:28.099592 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:28.099548    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:28.280443 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:28.280406    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="50443b8360a359c88ba28256d2b13d3a5388b0d4a5641fc7b0ee4db2abc22a51" exitCode=0
Nov 03 00:40:28.281268 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:28.280501    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"50443b8360a359c88ba28256d2b13d3a5388b0d4a5641fc7b0ee4db2abc22a51"}
Nov 03 00:40:29.098860 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:29.098673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:29.099006 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:29.098943    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:30.098494 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:30.098458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:30.098937 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:30.098601    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:30.098937 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:30.098656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:30.098937 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:30.098780    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:31.098998 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.098962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:31.099523 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.099065    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:31.289422 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.289391    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="eecf213bd7b63fa7aa991d39fbf37bb5a97b360fbd5e78208941358c85fa386e" exitCode=0
Nov 03 00:40:31.289581 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.289443    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"eecf213bd7b63fa7aa991d39fbf37bb5a97b360fbd5e78208941358c85fa386e"}
Nov 03 00:40:31.690916 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.690829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:31.691080 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.690968    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:31.691080 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.691024    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.691009986 +0000 UTC m=+34.179636415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:40:31.792146 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.792121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:31.792255 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:31.792181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:31.792321 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792251    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:31.792321 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792302    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret podName:04c2c55c-c14f-438b-b025-0ccf42d1d253 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.792288323 +0000 UTC m=+34.280914755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret") pod "global-pull-secret-syncer-7vg7f" (UID: "04c2c55c-c14f-438b-b025-0ccf42d1d253") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:40:31.792321 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792307    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:40:31.792461 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792326    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:40:31.792461 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792337    2578 projected.go:196] Error preparing data for projected volume kube-api-access-6kn7s for pod openshift-network-diagnostics/network-check-target-9g7gh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:31.792461 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:31.792394    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s podName:2ee31d79-9466-44c0-93e5-b69d44ebaf0b nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.792376196 +0000 UTC m=+34.281002625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kn7s" (UniqueName: "kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s") pod "network-check-target-9g7gh" (UID: "2ee31d79-9466-44c0-93e5-b69d44ebaf0b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:40:32.098490 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.098311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:32.098633 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.098359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:32.098633 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:32.098592    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:32.098748 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:32.098686    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:32.293882 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.293801    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"c7155994789b655c1fcb8e86b678f9a236dc07fbef1b601e4c3ff21057b5f05e"}
Nov 03 00:40:32.293882 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.293875    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"b975efbde799d8f9721f92ef5e0b5cb8991e6fa525a555edb61533e0173e1633"}
Nov 03 00:40:32.294326 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.293911    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"c9f9f50a1927338443f7f658bfb0569b12d71fa2c960781b3186a80c22a25f3d"}
Nov 03 00:40:32.294326 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.293948    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"112d3a76eddd3fd19e80bafbb7fa9a0af7fc6be80d40b9d4672cbf89ca1aaabf"}
Nov 03 00:40:32.294326 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.293976    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"743f77d0be307e750773cb314499558a814715bdb039aa1020038f06a67721f4"}
Nov 03 00:40:32.294326 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.294009    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"a0796017b227e237d4a5ba62ff0d5124f3f1896747e1518e76a0e276c349128d"}
Nov 03 00:40:32.296826 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.296797    2578 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="76d95a76252b845e4908e57e5705f694ba0a599d14c249659437f2917a1b2022" exitCode=0
Nov 03 00:40:32.296939 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:32.296841    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerDied","Data":"76d95a76252b845e4908e57e5705f694ba0a599d14c249659437f2917a1b2022"}
Nov 03 00:40:33.098772 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:33.098741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:33.098971 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:33.098849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:33.301976 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:33.301940    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cw99z" event={"ID":"a548ec83-53fc-4a5c-a5d8-574b6ee5e971","Type":"ContainerStarted","Data":"cdeeb2c37a948f64d9f66d0b69ed131b354755fd6069674a2072286fdf892f47"}
Nov 03 00:40:33.324398 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:33.324357    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cw99z" podStartSLOduration=7.69574869 podStartE2EDuration="19.324345781s" totalImagesPullingTime="11.628597091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:33.322773946 +0000 UTC m=+19.811400395" watchObservedRunningTime="2026-11-03 00:40:33.324345781 +0000 UTC m=+19.812972230"
Nov 03 00:40:34.099689 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:34.099656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:34.099842 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:34.099785    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:34.099842 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:34.099823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:34.099966 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:34.099880    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:34.306762 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:34.306729    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"bf99ea6b90ee6e9074c5305c8934d1c4af66d3784ae0968ac0e78fc373030f90"}
Nov 03 00:40:35.098491 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:35.098458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:35.098638 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:35.098561    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:36.098670 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.098635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:36.099141 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.098681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:36.099141 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:36.098761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:36.099141 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:36.098867    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:36.316080 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.316043    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" event={"ID":"e7044cbc-8ac5-4ada-a632-8011de51e2d9","Type":"ContainerStarted","Data":"9c91ce49fae9130b3b73a4b8bee9630254ffbd86c17b4793ae0bf5c71cf1abaf"}
Nov 03 00:40:36.316427 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.316405    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:36.316497 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.316481    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:36.316557 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.316529    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:36.331845 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.331816    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:36.332122 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.332099    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:36.350500 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:36.350384    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v" podStartSLOduration=6.904995058 podStartE2EDuration="22.350366847s" totalImagesPullingTime="15.445371789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:36.348298237 +0000 UTC m=+22.836924686" watchObservedRunningTime="2026-11-03 00:40:36.350366847 +0000 UTC m=+22.838993299"
Nov 03 00:40:37.099094 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.099058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:37.099622 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:37.099172    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:37.923317 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.923248    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9g7gh"]
Nov 03 00:40:37.923551 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.923350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:37.923551 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:37.923427    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:37.927215 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.927190    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vg7f"]
Nov 03 00:40:37.927312 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.927289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:37.927391 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:37.927370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:37.928885 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.928864    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlbmn"]
Nov 03 00:40:37.928971 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:37.928951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:37.929050 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:37.929033    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:39.098870 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:39.098840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:39.099321 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:39.098938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:40.098758 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:40.098727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:40.098898 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:40.098834    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:40.099253 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:40.098901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:40.099253 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:40.098983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:41.098606 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:41.098540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:41.098731 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:41.098642    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:42.098830 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:42.098792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:42.099206 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:42.098926    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:42.099206 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:42.098971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:42.099206 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:42.099061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:43.098358 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:43.098315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:43.098526 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:43.098425    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbmn" podUID="b7d495ba-cbe7-4415-b87d-968ead7c09a3"
Nov 03 00:40:44.099816 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.099659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:44.100179 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.099723    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:44.100179 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:44.099875    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9g7gh" podUID="2ee31d79-9466-44c0-93e5-b69d44ebaf0b"
Nov 03 00:40:44.100179 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:44.099961    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vg7f" podUID="04c2c55c-c14f-438b-b025-0ccf42d1d253"
Nov 03 00:40:44.846851 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.846822    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeReady"
Nov 03 00:40:44.846985 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.846939    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 00:40:44.914307 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.914228    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hxbss"]
Nov 03 00:40:44.950997 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.950968    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dcrmw"]
Nov 03 00:40:44.951171 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.951086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:44.962580 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.962561    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:40:44.962683 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.962613    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:40:44.962818 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.962803    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\""
Nov 03 00:40:44.962882 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.962870    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:40:44.965961 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.965948    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcrmw"]
Nov 03 00:40:44.966055 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.966041    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxbss"]
Nov 03 00:40:44.966087 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.966080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:44.975214 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.975195    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:40:44.975305 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.975255    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:40:44.975305 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:44.975256    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\""
Nov 03 00:40:45.087452 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.087452 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.087652 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fkp\" (UniqueName: \"kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.087652 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.087652 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2cjr\" (UniqueName: \"kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.087766 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.087658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.098520 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.098496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:45.105572 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.105552    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:40:45.105904 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.105890    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\""
Nov 03 00:40:45.188146 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fkp\" (UniqueName: \"kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.188321 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.188321 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cjr\" (UniqueName: \"kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.188321 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.188321 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.188321 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.188556 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:45.188339    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:40:45.188556 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:45.188425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls podName:6a6f16d7-d3db-4415-bf36-c05b4200e860 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:45.688402476 +0000 UTC m=+32.177028904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls") pod "dns-default-dcrmw" (UID: "6a6f16d7-d3db-4415-bf36-c05b4200e860") : secret "dns-default-metrics-tls" not found
Nov 03 00:40:45.188556 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188539    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.188854 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.188833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.192233 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.192216    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.202689 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.202655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fkp\" (UniqueName: \"kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.202828 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.202813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cjr\" (UniqueName: \"kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.272290 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.272258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:45.616781 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.616750    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxbss"]
Nov 03 00:40:45.620454 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:45.620428    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ea7fa2_8908_48c9_a0d5_e10ca4dbc3e7.slice/crio-10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca WatchSource:0}: Error finding container 10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca: Status 404 returned error can't find the container with id 10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca
Nov 03 00:40:45.692735 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:45.692700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:45.692884 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:45.692819    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:40:45.692921 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:45.692884    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls podName:6a6f16d7-d3db-4415-bf36-c05b4200e860 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:46.692867363 +0000 UTC m=+33.181493798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls") pod "dns-default-dcrmw" (UID: "6a6f16d7-d3db-4415-bf36-c05b4200e860") : secret "dns-default-metrics-tls" not found
Nov 03 00:40:46.101567 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.101535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:46.101567 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.101566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:46.106246 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.106221    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:40:46.106529 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.106312    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\""
Nov 03 00:40:46.109373 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.109354    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:40:46.206484 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.206446    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:40:46.206623 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.206525    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\""
Nov 03 00:40:46.339483 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.339448    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxbss" event={"ID":"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7","Type":"ContainerStarted","Data":"10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca"}
Nov 03 00:40:46.656532 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.656500    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"]
Nov 03 00:40:46.678640 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.678615    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"]
Nov 03 00:40:46.678799 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.678728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:46.697279 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.697257    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:46.697725 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.697705    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:40:46.698605 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.698582    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dx4w2\""
Nov 03 00:40:46.698713 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.698694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:46.698803 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:46.698790    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:40:46.698852 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:46.698841    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls podName:6a6f16d7-d3db-4415-bf36-c05b4200e860 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.698827291 +0000 UTC m=+35.187453738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls") pod "dns-default-dcrmw" (UID: "6a6f16d7-d3db-4415-bf36-c05b4200e860") : secret "dns-default-metrics-tls" not found
Nov 03 00:40:46.786608 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.786568    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:40:46.799390 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.799340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:46.799547 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.799408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxmf\" (UniqueName: \"kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf\") pod \"migrator-66df5d496b-st6fr\" (UID: \"ee642aa1-5652-4b69-acd8-0fb5c48cd430\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:46.803459 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.802429    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8x65q"]
Nov 03 00:40:46.817429 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.817406    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:40:46.817521 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.817511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.824202 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.823699    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x65q"]
Nov 03 00:40:46.890071 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.890033    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:40:46.890271 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.890237    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:40:46.890271 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.890237    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:40:46.890271 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.890237    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:40:46.896263 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.896239    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:40:46.896398 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.896332    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tg82s\""
Nov 03 00:40:46.896398 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.896325    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\""
Nov 03 00:40:46.896398 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.896332    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:40:46.900577 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:46.900694 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.900694 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.900694 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq85c\" (UniqueName: \"kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.900865 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:46.900865 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.900865 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:46.900865 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.900835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxmf\" (UniqueName: \"kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf\") pod \"migrator-66df5d496b-st6fr\" (UID: \"ee642aa1-5652-4b69-acd8-0fb5c48cd430\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:46.901037 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.901022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:46.901097 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.901052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:46.910012 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:46.909965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxmf\" (UniqueName: \"kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf\") pod \"migrator-66df5d496b-st6fr\" (UID: \"ee642aa1-5652-4b69-acd8-0fb5c48cd430\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:47.001748 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq85c\" (UniqueName: \"kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.001928 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.001968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.002069    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.002101    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.002116    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.502102248 +0000 UTC m=+33.990728678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:47.002234 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.002183    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:47.502151262 +0000 UTC m=+33.990777705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : configmap "kube-rbac-proxy" not found
Nov 03 00:40:47.002596 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:47.002596 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002411    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.002596 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.002786 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.002893 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.002874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.011012 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.010988    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq85c\" (UniqueName: \"kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.011259 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.011241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.122280 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.122249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:47.287824 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.287791    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"]
Nov 03 00:40:47.290441 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:47.290410    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee642aa1_5652_4b69_acd8_0fb5c48cd430.slice/crio-858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266 WatchSource:0}: Error finding container 858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266: Status 404 returned error can't find the container with id 858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266
Nov 03 00:40:47.343619 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.343584    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr" event={"ID":"ee642aa1-5652-4b69-acd8-0fb5c48cd430","Type":"ContainerStarted","Data":"858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266"}
Nov 03 00:40:47.381296 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.381261    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvtfb"]
Nov 03 00:40:47.399094 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.399068    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvtfb"]
Nov 03 00:40:47.399274 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.399214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.409980 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.409862    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 00:40:47.409980 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.409913    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 00:40:47.409980 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.409931    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 00:40:47.410173 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.409996    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4c8qt\""
Nov 03 00:40:47.410173 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.410070    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 00:40:47.412149 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.412128    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:40:47.416436 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:47.416408    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ebb569_3603_406a_9999_42f66e578212.slice/crio-e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0 WatchSource:0}: Error finding container e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0: Status 404 returned error can't find the container with id e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0
Nov 03 00:40:47.506828 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.506748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.506828 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.506782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47gd\" (UniqueName: \"kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.506828 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.506809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.507096 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.506938    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:47.507096 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.507009    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.50699018 +0000 UTC m=+34.995616609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:47.507233 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.507146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:47.507233 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.507214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.507333 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.507255    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:40:47.507383 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.507341    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:48.50731401 +0000 UTC m=+34.995940453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : configmap "kube-rbac-proxy" not found
Nov 03 00:40:47.607838 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.607805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.608015 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.607864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47gd\" (UniqueName: \"kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.608015 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.607881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.608517 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.608481    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.610474 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.610452    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.617917 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.617896    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47gd\" (UniqueName: \"kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.709210 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.709178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:47.709365 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.709346    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:40:47.709415 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:47.709411    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs podName:b7d495ba-cbe7-4415-b87d-968ead7c09a3 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:19.709394709 +0000 UTC m=+66.198021137 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs") pod "network-metrics-daemon-rlbmn" (UID: "b7d495ba-cbe7-4415-b87d-968ead7c09a3") : secret "metrics-daemon-secret" not found
Nov 03 00:40:47.726024 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.725998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:47.810421 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.809723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:47.810421 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.809760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:47.812603 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.812190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:47.812750 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.812729    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:47.936694 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.936443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:47.951954 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:47.951786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:48.018460 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.018330    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xvtfb"]
Nov 03 00:40:48.026920 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:48.026286    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8d638b_12ea_4bf4_b6a2_ec9b4107876f.slice/crio-0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63 WatchSource:0}: Error finding container 0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63: Status 404 returned error can't find the container with id 0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63
Nov 03 00:40:48.238285 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.238053    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9g7gh"]
Nov 03 00:40:48.271089 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.271060    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vg7f"]
Nov 03 00:40:48.275798 ip-10-0-136-112 kubenswrapper[2578]: W1103 00:40:48.275771    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c2c55c_c14f_438b_b025_0ccf42d1d253.slice/crio-9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871 WatchSource:0}: Error finding container 9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871: Status 404 returned error can't find the container with id 9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871
Nov 03 00:40:48.348190 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.348140    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb" event={"ID":"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f","Type":"ContainerStarted","Data":"0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63"}
Nov 03 00:40:48.349791 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.349742    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9g7gh" event={"ID":"2ee31d79-9466-44c0-93e5-b69d44ebaf0b","Type":"ContainerStarted","Data":"cdf32038a377767e914e48ff0bd12686c4dfa8bbfec9a77ca0d830874927da19"}
Nov 03 00:40:48.351170 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.351128    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vg7f" event={"ID":"04c2c55c-c14f-438b-b025-0ccf42d1d253","Type":"ContainerStarted","Data":"9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871"}
Nov 03 00:40:48.352465 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.352445    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" event={"ID":"27ebb569-3603-406a-9999-42f66e578212","Type":"ContainerStarted","Data":"e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0"}
Nov 03 00:40:48.515333 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.515294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:48.515557 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.515375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:48.515557 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.515460    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:48.515557 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.515484    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:40:48.515557 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.515535    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:50.515520362 +0000 UTC m=+37.004146806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:48.515557 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.515551    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:50.515545125 +0000 UTC m=+37.004171569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : configmap "kube-rbac-proxy" not found
Nov 03 00:40:48.721180 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:48.717300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:48.721180 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.717443    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:40:48.721180 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:48.717500    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls podName:6a6f16d7-d3db-4415-bf36-c05b4200e860 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:52.71748356 +0000 UTC m=+39.206110011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls") pod "dns-default-dcrmw" (UID: "6a6f16d7-d3db-4415-bf36-c05b4200e860") : secret "dns-default-metrics-tls" not found
Nov 03 00:40:49.358689 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:49.358650    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxbss" event={"ID":"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7","Type":"ContainerStarted","Data":"1809cb388641de300ede8d8cdcf74700cb363ab09fc058c569d883af825c25cc"}
Nov 03 00:40:49.361387 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:49.361355    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr" event={"ID":"ee642aa1-5652-4b69-acd8-0fb5c48cd430","Type":"ContainerStarted","Data":"a1f2f4992b3177ada773db01e44462cdfaa8edf6d31efd8dbe69365a77922a60"}
Nov 03 00:40:49.387731 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:49.387674    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hxbss" podStartSLOduration=2.366472713 podStartE2EDuration="5.38765536s" totalImagesPullingTime="3.021182647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:49.38737235 +0000 UTC m=+35.875998802" watchObservedRunningTime="2026-11-03 00:40:49.38765536 +0000 UTC m=+35.876281812"
Nov 03 00:40:49.409074 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:49.409020    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr" podStartSLOduration=1.739899273 podStartE2EDuration="3.409000751s" totalImagesPullingTime="1.669101478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:49.406501027 +0000 UTC m=+35.895127478" watchObservedRunningTime="2026-11-03 00:40:49.409000751 +0000 UTC m=+35.897627203"
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:50.535816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:50.535904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:50.535971    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:50.536026    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.536010005 +0000 UTC m=+41.024636448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:50.536363    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:40:50.549244 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:50.536412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:54.536396801 +0000 UTC m=+41.025023244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : configmap "kube-rbac-proxy" not found
Nov 03 00:40:51.369478 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:51.369437    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" event={"ID":"27ebb569-3603-406a-9999-42f66e578212","Type":"ContainerStarted","Data":"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"}
Nov 03 00:40:51.754794 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:51.754717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbkz7_3c45381d-5957-4d81-847f-5e3ad1b1780c/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:52.374123 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.374019    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb" event={"ID":"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f","Type":"ContainerStarted","Data":"f90721ba555deb9cc56ceb42b849dc9a9c72a3043c782d154e95cbb012b6c2c7"}
Nov 03 00:40:52.393944 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.393886    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb" podStartSLOduration=1.7249046300000002 podStartE2EDuration="5.393869273s" totalImagesPullingTime="3.668964643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:52.393536593 +0000 UTC m=+38.882163044" watchObservedRunningTime="2026-11-03 00:40:52.393869273 +0000 UTC m=+38.882495723"
Nov 03 00:40:52.394391 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.394358    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" podStartSLOduration=3.326685725 podStartE2EDuration="6.394348479s" totalImagesPullingTime="3.067662754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:51.387338062 +0000 UTC m=+37.875964516" watchObservedRunningTime="2026-11-03 00:40:52.394348479 +0000 UTC m=+38.882974931"
Nov 03 00:40:52.690091 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.690005    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbkz7_3c45381d-5957-4d81-847f-5e3ad1b1780c/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:52.756358 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.756171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:52.756358 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:52.756297    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:40:52.756358 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:52.756356    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls podName:6a6f16d7-d3db-4415-bf36-c05b4200e860 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:00.756341463 +0000 UTC m=+47.244967914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls") pod "dns-default-dcrmw" (UID: "6a6f16d7-d3db-4415-bf36-c05b4200e860") : secret "dns-default-metrics-tls" not found
Nov 03 00:40:52.825179 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:52.824250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbkz7_3c45381d-5957-4d81-847f-5e3ad1b1780c/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:53.378760 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:53.378721    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9g7gh" event={"ID":"2ee31d79-9466-44c0-93e5-b69d44ebaf0b","Type":"ContainerStarted","Data":"bfc7b08a7f0fc917f9c4807734db0b9c2f3791d21e15a675b425f738f74d6990"}
Nov 03 00:40:53.378933 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:53.378808    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:53.397639 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:53.397586    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9g7gh" podStartSLOduration=35.094544881 podStartE2EDuration="39.397566425s" totalImagesPullingTime="4.303021544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:53.396391873 +0000 UTC m=+39.885018323" watchObservedRunningTime="2026-11-03 00:40:53.397566425 +0000 UTC m=+39.886192876"
Nov 03 00:40:53.783349 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:53.783267    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wrzvg_2bfad136-a64a-4525-ae3a-d77fb5af1084/dns-node-resolver/0.log"
Nov 03 00:40:54.575122 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:54.575094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:54.575267 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:54.575223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:54.575372 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:54.575356    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:54.575436 ip-10-0-136-112 kubenswrapper[2578]: E1103 00:40:54.575425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:02.575407121 +0000 UTC m=+49.064033567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:54.575715 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:54.575693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:54.583342 ip-10-0-136-112 kubenswrapper[2578]: I1103 00:40:54.583317    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9wpw_1355bd78-3c21-4bf1-a585-8e66bf03cbb1/node-ca/0.log"
Nov 03 00:40:54.720418 ip-10-0-136-112 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 00:40:54.730634 ip-10-0-136-112 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 00:40:54.730801 ip-10-0-136-112 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 00:40:54.730847 ip-10-0-136-112 systemd[1]: kubelet.service: Consumed 2.345s CPU time.
Nov 03 00:40:54.740797 ip-10-0-136-112 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:40:54.825508 ip-10-0-136-112 kubenswrapper[4531]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:54.825508 ip-10-0-136-112 kubenswrapper[4531]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:40:54.825508 ip-10-0-136-112 kubenswrapper[4531]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:54.825508 ip-10-0-136-112 kubenswrapper[4531]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:40:54.827533 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827457    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.827533 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827533    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827537    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827541    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827544    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827546    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827549    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827552    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827556    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827559    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827562    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827564    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827567    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827569    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827572    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827575    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827578    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827580    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827583    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827586    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.827608 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827588    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827591    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827593    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827596    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827599    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827608    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827613    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827616    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827619    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827623    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827626    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827629    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827632    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827635    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827638    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827640    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827643    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827645    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827648    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.828085 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827652    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827656    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827659    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827662    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827664    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827667    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827670    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827672    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827675    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827677    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827681    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827684    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827687    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827690    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827693    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827696    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827698    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827701    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827704    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827706    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.828635 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827709    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827711    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827714    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827716    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827719    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827721    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827724    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827727    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827730    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827733    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827736    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827739    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827741    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827744    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827746    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827749    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827751    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827754    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827756    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827759    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.829126 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827761    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827764    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827767    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827769    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827772    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827775    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827777    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827780    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827782    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827785    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827787    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827790    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827792    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827795    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827797    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827800    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827802    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827805    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827808    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827810    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.829625 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827814    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827817    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827819    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827822    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827824    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827827    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.827829    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828627    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828635    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828637    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828641    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828643    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828647    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828650    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828653    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828655    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828658    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828661    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828664    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828666    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.830128 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828669    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828671    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828674    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828677    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828679    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828682    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828685    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828687    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828690    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828693    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828695    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828698    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828701    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828703    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828706    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828709    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828711    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828714    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828717    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.830653 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828719    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828722    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828725    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828727    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828730    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828733    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828739    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828742    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828745    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828748    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828751    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828754    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828756    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828759    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828762    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828764    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828767    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828769    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828772    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828774    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.831117 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828777    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828780    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828782    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828787    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828793    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828796    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828799    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828802    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828805    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828808    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828810    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828813    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828815    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828818    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828821    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828823    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828826    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828829    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828831    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.831614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828834    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828837    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828839    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828843    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828846    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828848    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828851    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828853    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828856    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828858    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828861    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828863    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828866    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828868    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828871    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828873    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828876    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828881    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828884    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828887    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.832087 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828890    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828892    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828895    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828897    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828900    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828903    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828905    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828908    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828910    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828913    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828917    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828920    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828923    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828926    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.828930    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829512    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829522    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829526    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829529    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.832604 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829532    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829535    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829537    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829540    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829542    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829545    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829548    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829551    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829553    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829555    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829560    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829563    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829566    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829568    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829571    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829573    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829576    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829579    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829582    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829584    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.833067 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829587    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829589    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829592    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829594    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829597    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829600    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829609    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829612    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829615    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829618    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829620    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829623    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829625    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829628    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829631    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829633    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829636    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829639    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829642    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.833603 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829644    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829646    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829649    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829652    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829656    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829659    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829662    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829664    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829667    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829669    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829672    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829675    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829677    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829680    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829683    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829685    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829688    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829690    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829693    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.834099 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829696    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829704    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829707    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829710    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829712    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829716    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829718    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829721    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829723    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829726    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829729    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829731    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829733    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829736    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829739    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829741    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829744    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829747    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829749    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829752    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.834588 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829755    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829757    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829760    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829762    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829765    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829768    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829770    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829773    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829775    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829778    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829781    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829784    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829786    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829789    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829798    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829801    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829803    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829806    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829808    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829811    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.835190 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829814    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829818    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829821    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.829824    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829916    4531 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829923    4531 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829934    4531 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829939    4531 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829944    4531 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829947    4531 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829951    4531 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829955    4531 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829958    4531 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829961    4531 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829965    4531 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829968    4531 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829971    4531 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829974    4531 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829977    4531 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829980    4531 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829983    4531 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829985    4531 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829993    4531 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:40:54.835690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.829996    4531 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830000    4531 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830003    4531 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830006    4531 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830016    4531 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830019    4531 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830022    4531 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830025    4531 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830028    4531 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830031    4531 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830034    4531 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830038    4531 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830040    4531 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830045    4531 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830048    4531 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830051    4531 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830054    4531 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830057    4531 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830060    4531 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830067    4531 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830070    4531 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830073    4531 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830075    4531 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830079    4531 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:40:54.836271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830084    4531 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830087    4531 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830091    4531 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830094    4531 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830097    4531 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830100    4531 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830103    4531 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830106    4531 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830110    4531 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830113    4531 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830115    4531 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830120    4531 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830123    4531 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830132    4531 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830136    4531 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830139    4531 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830142    4531 flags.go:64] FLAG: --help="false"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830145    4531 flags.go:64] FLAG: --hostname-override="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830148    4531 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830151    4531 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830154    4531 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830175    4531 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830181    4531 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:40:54.837206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830185    4531 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830190    4531 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830193    4531 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830195    4531 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830198    4531 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830202    4531 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830204    4531 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830214    4531 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830217    4531 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830222    4531 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830225    4531 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830227    4531 flags.go:64] FLAG: --lock-file=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830230    4531 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830233    4531 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830236    4531 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830242    4531 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830245    4531 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830249    4531 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830252    4531 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830255    4531 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830258    4531 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830261    4531 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830264    4531 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830274    4531 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830277    4531 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:40:54.837932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830281    4531 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830284    4531 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830287    4531 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830290    4531 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830293    4531 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830296    4531 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830299    4531 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830302    4531 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830309    4531 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830312    4531 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830315    4531 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830318    4531 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830321    4531 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830324    4531 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830327    4531 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830330    4531 flags.go:64] FLAG: --port="10250"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830335    4531 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830338    4531 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03440cf63280987ad"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830341    4531 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830344    4531 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830347    4531 flags.go:64] FLAG: --register-node="true"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830352    4531 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830356    4531 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830359    4531 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830361    4531 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830364    4531 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:40:54.838576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830368    4531 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830371    4531 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830374    4531 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830377    4531 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830380    4531 flags.go:64] FLAG: --runonce="false"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830391    4531 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830394    4531 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830398    4531 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830401    4531 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830403    4531 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830406    4531 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830410    4531 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830413    4531 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830416    4531 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830419    4531 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830422    4531 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830425    4531 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830428    4531 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830431    4531 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830434    4531 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830439    4531 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830442    4531 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830446    4531 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830453    4531 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830456    4531 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:40:54.839212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830458    4531 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830461    4531 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830466    4531 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830469    4531 flags.go:64] FLAG: --v="2"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830473    4531 flags.go:64] FLAG: --version="false"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830477    4531 flags.go:64] FLAG: --vmodule=""
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830484    4531 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.830487    4531 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830683    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830688    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830692    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830695    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830698    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830706    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830709    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830712    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830714    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830717    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830719    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830722    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830725    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.839824 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830727    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830730    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830732    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830735    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830738    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830740    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830743    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830745    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830750    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830752    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830755    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830758    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830761    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830765    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830767    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830770    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830773    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830775    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830778    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830780    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.840434 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830783    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830785    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830788    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830791    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830793    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830801    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830804    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830807    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830809    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830812    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830815    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830817    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830820    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830823    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830825    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830828    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830830    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830833    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830835    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.840931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830838    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830842    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830844    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830847    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830849    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830852    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830856    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830859    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830861    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830864    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830866    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830869    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830872    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830875    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830877    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830880    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830882    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830885    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830888    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830891    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.841406 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830894    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830897    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830899    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830902    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830905    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830907    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830910    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830913    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830915    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830918    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830920    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830923    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830925    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830929    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830932    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830934    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830939    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830942    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830946    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.842041 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830949    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830951    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830954    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830957    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830959    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830962    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830964    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830967    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830970    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830972    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830975    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830977    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830980    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830984    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.830988    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.831015    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:54.842620 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831060    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831065    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831069    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831072    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831075    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831078    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831081    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831083    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831086    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831089    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831091    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831095    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831098    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831100    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831103    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831107    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831110    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831113    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831115    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831118    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.843051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831120    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831123    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831126    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831129    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831131    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831134    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831136    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831139    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831142    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831145    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831148    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831150    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831153    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831156    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831170    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831172    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831175    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831178    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831180    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.843609 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831184    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831188    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831191    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831194    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831198    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831201    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831204    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831206    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831211    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831214    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831216    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831219    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831222    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831224    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831227    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831230    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831234    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831238    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831241    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831244    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.844184 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831247    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831250    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831253    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831256    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831259    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831261    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831264    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831267    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831269    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831272    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831274    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831277    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831280    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831282    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831285    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831287    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831292    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831294    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831297    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.844795 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831299    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831303    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831306    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831309    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831312    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831315    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831317    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831320    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831322    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831325    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831328    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831330    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831333    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831335    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831338    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831340    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831343    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831346    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831349    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.845396 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831351    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831354    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831356    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831359    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831362    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831364    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831368    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831370    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.831373    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.835735    4531 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.835756    4531 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.835782    4531 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.835789    4531 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835870    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835877    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835881    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835886    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835891    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835895    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.845862 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835900    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835904    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835908    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835912    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835916    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835921    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835925    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835930    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835934    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835938    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835943    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835947    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835951    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835956    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835960    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835967    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835974    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835978    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835982    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.846336 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835987    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835991    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.835996    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836000    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836005    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836010    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836014    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836018    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836022    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836026    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836030    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836034    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836038    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836043    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836047    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836051    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836055    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836059    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836063    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836067    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.846798 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836071    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836076    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836080    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836084    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836088    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836092    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836096    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836100    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836112    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836116    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836120    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836125    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836129    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836133    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836137    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836141    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836146    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836150    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836154    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836172    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.847338 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836177    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836180    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836185    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836189    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836193    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836197    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836201    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836205    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836209    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836213    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836217    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836221    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836225    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836230    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836234    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836238    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836243    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836247    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836252    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836256    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.848026 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836260    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836264    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836268    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836273    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836280    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836285    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836291    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836296    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836301    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836305    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836309    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836313    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836318    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836321    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836326    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836330    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836334    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836338    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.848858 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836342    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836346    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.836350    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.836388    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837120    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837130    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837135    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837139    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837145    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837149    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837173    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837177    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837182    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837187    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837191    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837195    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.849334 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837200    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837204    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837209    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837213    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837218    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837222    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837231    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837236    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837240    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837245    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837249    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837253    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837258    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837262    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837266    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837270    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837274    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837278    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837287    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837291    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.849749 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837295    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837300    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837304    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837310    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837317    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837322    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837326    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837330    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837334    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837338    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837343    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837352    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837356    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837360    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837364    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837368    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837373    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837377    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837381    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.850233 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837385    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837390    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837394    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837398    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837403    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837412    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837416    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837420    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837424    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837429    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837433    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837437    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837441    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837445    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837450    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837454    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837458    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837468    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837472    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.850744 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837476    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837481    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837485    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837489    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837495    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837500    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837504    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.837928    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838074    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838079    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838082    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838086    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838089    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838092    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838097    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838103    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838106    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838109    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838117    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838119    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.851220 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838123    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838126    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838128    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838131    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838134    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838137    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838140    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838142    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838145    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838147    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838150    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838153    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838156    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838179    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838184    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838186    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.851693 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.838231    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838272    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838277    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838282    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838287    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838290    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838293    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838296    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838298    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838301    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838304    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838307    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838309    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838312    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838315    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838318    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838321    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838323    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838326    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:40:54.852127 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838329    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838331    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838334    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838337    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838340    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838343    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838345    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838348    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838351    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838353    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838356    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838358    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838361    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838363    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838366    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838369    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838371    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838374    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838377    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:40:54.852693 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838379    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838382    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838384    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838387    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838389    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838392    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838395    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838397    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838400    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838402    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838405    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838408    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838410    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838413    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838416    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838420    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838423    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838426    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838429    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838432    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:40:54.853199 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838434    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838437    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838439    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838442    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838445    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838447    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838450    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838453    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838455    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838458    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838461    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838464    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838466    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838479    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838482    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838485    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838489    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838493    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838498    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:40:54.853940 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838502    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838505    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838507    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838510    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838512    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838515    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838519    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838522    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838526    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838531    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838533    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838536    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838539    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838542    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838545    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838547    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838551    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838554    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838557    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838559    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:40:54.854628 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838562    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838564    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838567    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838570    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838572    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838575    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838577    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838580    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838583    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:54.838586    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.838644    4531 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.839636    4531 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.839686    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.839850    4531 server.go:1014] "Starting client certificate rotation"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.840000    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.840058    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 00:35:14 +0000 UTC" deadline="2028-08-20 12:27:57.940638052 +0000 UTC"
Nov 03 00:40:54.855108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.840095    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15755h47m3.100546279s"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.840338    4531 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.841427    4531 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.842718    4531 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.844260    4531 log.go:25] "Validated CRI v1 image API"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.844956    4531 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:40:54.855590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.846469    4531 fs.go:135] Filesystem UUIDs: map[1bb74329-46d0-487e-ba8a-1297fbf39683:/dev/nvme0n1p4 36e90348-92e9-40ca-83ca-03c399f27cd3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 00:40:54.855799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.846488    4531 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdf32038a377767e914e48ff0bd12686c4dfa8bbfec9a77ca0d830874927da19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdf32038a377767e914e48ff0bd12686c4dfa8bbfec9a77ca0d830874927da19/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04c2c55c-c14f-438b-b025-0ccf42d1d253/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/04c2c55c-c14f-438b-b025-0ccf42d1d253/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1355bd78-3c21-4bf1-a585-8e66bf03cbb1/volumes/kubernetes.io~projected/kube-api-access-pwz8n:{mountpoint:/var/lib/kubelet/pods/1355bd78-3c21-4bf1-a585-8e66bf03cbb1/volumes/kubernetes.io~projected/kube-api-access-pwz8n major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27ebb569-3603-406a-9999-42f66e578212/volumes/kubernetes.io~projected/kube-api-access-tbdcc:{mountpoint:/var/lib/kubelet/pods/27ebb569-3603-406a-9999-42f66e578212/volumes/kubernetes.io~projected/kube-api-access-tbdcc major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bfad136-a64a-4525-ae3a-d77fb5af1084/volumes/kubernetes.io~projected/kube-api-access-wrn8d:{mountpoint:/var/lib/kubelet/pods/2bfad136-a64a-4525-ae3a-d77fb5af1084/volumes/kubernetes.io~projected/kube-api-access-wrn8d major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee31d79-9466-44c0-93e5-b69d44ebaf0b/volumes/kubernetes.io~projected/kube-api-access-6kn7s:{mountpoint:/var/lib/kubelet/pods/2ee31d79-9466-44c0-93e5-b69d44ebaf0b/volumes/kubernetes.io~projected/kube-api-access-6kn7s major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c45381d-5957-4d81-847f-5e3ad1b1780c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3c45381d-5957-4d81-847f-5e3ad1b1780c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~projected/kube-api-access-c2cjr:{mountpoint:/var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~projected/kube-api-access-c2cjr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a6f16d7-d3db-4415-bf36-c05b4200e860/volumes/kubernetes.io~projected/kube-api-access-c4fkp:{mountpoint:/var/lib/kubelet/pods/6a6f16d7-d3db-4415-bf36-c05b4200e860/volumes/kubernetes.io~projected/kube-api-access-c4fkp major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eec9d2b-a414-4463-9745-5a9dc01f4061/volumes/kubernetes.io~projected/kube-api-access-fskw4:{mountpoint:/var/lib/kubelet/pods/7eec9d2b-a414-4463-9745-5a9dc01f4061/volumes/kubernetes.io~projected/kube-api-access-fskw4 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91635f66-bcd2-4510-ad4b-d0412ace3255/volumes/kubernetes.io~projected/kube-api-access-5972d:{mountpoint:/var/lib/kubelet/pods/91635f66-bcd2-4510-ad4b-d0412ace3255/volumes/kubernetes.io~projected/kube-api-access-5972d major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a548ec83-53fc-4a5c-a5d8-574b6ee5e971/volumes/kubernetes.io~projected/kube-api-access-hjp97:{mountpoint:/var/lib/kubelet/pods/a548ec83-53fc-4a5c-a5d8-574b6ee5e971/volumes/kubernetes.io~projected/kube-api-access-hjp97 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7d495ba-cbe7-4415-b87d-968ead7c09a3/volumes/kubernetes.io~projected/kube-api-access-bj9m6:{mountpoint:/var/lib/kubelet/pods/b7d495ba-cbe7-4415-b87d-968ead7c09a3/volumes/kubernetes.io~projected/kube-api-access-bj9m6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~projected/kube-api-access-w47gd:{mountpoint:/var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~projected/kube-api-access-w47gd major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~projected/kube-api-access-vcgcw:{mountpoint:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~projected/kube-api-access-vcgcw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d/volumes/kubernetes.io~projected/kube-api-access-24xfn:{mountpoint:/var/lib/kubelet/pods/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d/volumes/kubernetes.io~projected/kube-api-access-24xfn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~projected/kube-api-access-4vpp4:{mountpoint:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~projected/kube-api-access-4vpp4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee642aa1-5652-4b69-acd8-0fb5c48cd430/volumes/kubernetes.io~projected/kube-api-access-hzxmf:{mountpoint:/var/lib/kubelet/pods/ee642aa1-5652-4b69-acd8-0fb5c48cd430/volumes/kubernetes.io~projected/kube-api-access-hzxmf major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc1e3ecf-7531-454d-93a5-5a6cce661c27/volumes/kubernetes.io~projected/kube-api-access-wq85c:{mountpoint:/var/lib/kubelet/pods/fc1e3ecf-7531-454d-93a5-5a6cce661c27/volumes/kubernetes.io~projected/kube-api-access-wq85c major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a31e17efdf06e30b50d79af8870b833d52dae5ed4dc270fc1bfbbc6a73d2ce30/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b0b0909c041522b50bd52ad857bac50452744cae653b12ec60c5e696d5cdb889/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/262a08d587ca44a5fe995f5b387a790c767b861669dc57bbd2a116f635f06ffd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/15411dc6e4d8be5ceece5b1ca11048d9ee403278d6fbf75e51cc6797b6176f37/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1cfc76e256ea7b2706c1c0a5e7bde176fb8c543328eb1f3c615ffded91a1cba0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3871a4c8f0ab81b6ff6a854e070ac6110bd48997bb9fec540b9937cd2a8c6ae7/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1a2786322e2cbe56db449223fa7d5ed71569b9e8173778fcc9a484afffeb5eee/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/918d658c4c5f6e7d10e3d640ac689e04bb86b258eb09554cc2adc04b635338ec/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a36897f46337494721bcf541b91252df13c46eb1bfcb7f5c79511dcfd534ccb0/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/1cd1ae8ecf341306f3078d0826ef77f81c3616b43077e94570cd5c83232beea7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/37e69e5b40d11333f23beaa7511b3f2cbc26c7cf724002afd3f336610745a906/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/296b231b97a8a8735b07785ae27087e7c19c148a598a1e6aa9ac9a6948eb6944/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fa581b20296091bc9143c261b4bd6e01c916b2c81d7c530ae286d02cea4c6c7f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0f5c2e2e179821af6ed055b76bd5cd558b7e9d8434c6d941f00b2fe19cb7f209/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/11e2875bf1c09377150619993ac6172b9bea88c8a9b283cff2a6c58f80586ae5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3ba98d4a7db575554238933121466217fd5d6a9d5971bc95d84915482f968d87/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/03afed1524d0133edfc135efd3ec96288e8a0c9a49c7bcbed64e789653be8ca9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b50241e06814fb5a37cdd1e5c97c3551f9039186edd415e26c73fab6f88b618b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/52c2d3d302940a6c71a93f4f1a9c7214240dbb2236df73802592e389f8bf0503/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e48d6f1a34a8e53966fdbca6d1760b54b10ed97d614754e61bc6d73736667ae0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/09d91e9e2d7143face46f8d651b027a68bca143b9e917e5454729e8814065b4d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/74f8412e2a568b5dd1002825a394df82a4e69b70727f61547591e89c906cfce1/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/86de7d5c7cce6b97595b00cf362141154ad7253f62e2df08bfb413206382701b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/dd9d57788c74e9267d69b7edcc5ba6113a9dc328045f4d1f51f523b05b692fe3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/b8410d35ddf33b639cb3ec033deb29a78a252e1132a77bb004ef203ed7b8e519/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/2b3fa6021e3e694cc7b2c1625a5a0c27924fa0d586d85c363cc8c571f9e6de6b/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/fc9a4bd10e7bacaef5d1a4a796e3c134337a00fdbaec7e6170d0f99d4560c8b5/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/3a13ffce45f20a55be3a00c48ea5d7ae8dd58a874d5a0301a14b6179c1eb380f/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/73264219d877ce0ed987b1d26102d1c9015d618a1ea76b75ac46a6ade829a362/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/51a00c53626f9d67a2ae857f92a52940bfc6365bca9c5b6fd861706346a3bd42/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/66586a850d97202c1110b8c0cc77f41490ec13aa35a31246c87ccb07d0a73680/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e34ff04ef6eb25aa559f49e28e1bfad94cf30bb2d0de88273a76270e0729d8d9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/5779a9f6d6c074c8770c44655d244ee848c30498de70f6116fa4c92d1f1255b4/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/c6ff49e785c51ef6cca1ec7b868769327dfeaf1f4f1cbdebec75f2625bdc8c13/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5fb8997a73beee52b16b68617e5918483d534cd546ab6786b9e86cf538005008/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3fef736b1b495229867f277756c7b2d480af676f69f1461aeaee0ae7dcbf8590/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7696bf8adfab13248d6bb9a85add57a09e840da7f80d09f26d697abf04421505/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9f51b401bc3a585bc8a868747dd6b0ed8565a384192f0ee8add535c37bfdd99c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c70f99056f5014722204b556434399453732d4c0763f151f315f41225ce9c74c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5b7248dff249dce1338c4afd50f64987a0a5308cd17d40659e666f9fa6d1f5d5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4b515dc2f275a6801a70085e12d707469e505c5a11cba69e3694d003fdec986b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/ba7b4ae1d48d8ec9326881ed24ce01181722a443ef3649c01071cb50b2038a54/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c6e319d9ba7ade73ff29e176d06f8b44988f7ff5dfbb7cdcf4a41b493e6e5271/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.853026    4531 manager.go:217] Machine: {Timestamp:2026-11-03 00:40:54.852544297 +0000 UTC m=+0.066287960 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100481 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c9ed336111058ad4b8b38f43687d SystemUUID:ec28c9ed-3361-1105-8ad4-b8b38f43687d BootID:ddb5c592-1a7d-4413-9065-de3e8d70892d Filesystems:[{Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee31d79-9466-44c0-93e5-b69d44ebaf0b/volumes/kubernetes.io~projected/kube-api-access-6kn7s DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a548ec83-53fc-4a5c-a5d8-574b6ee5e971/volumes/kubernetes.io~projected/kube-api-access-hjp97 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10a3c69b21e3fa42e3b75a0da7cd53faf76ef184570556a16fa823b9e1c19fca/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fc1e3ecf-7531-454d-93a5-5a6cce661c27/volumes/kubernetes.io~projected/kube-api-access-wq85c DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdf32038a377767e914e48ff0bd12686c4dfa8bbfec9a77ca0d830874927da19/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~projected/kube-api-access-c2cjr DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c45381d-5957-4d81-847f-5e3ad1b1780c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91635f66-bcd2-4510-ad4b-d0412ace3255/volumes/kubernetes.io~projected/kube-api-access-5972d DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b7d495ba-cbe7-4415-b87d-968ead7c09a3/volumes/kubernetes.io~projected/kube-api-access-bj9m6 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27ebb569-3603-406a-9999-42f66e578212/volumes/kubernetes.io~projected/kube-api-access-tbdcc DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bfad136-a64a-4525-ae3a-d77fb5af1084/volumes/kubernetes.io~projected/kube-api-access-wrn8d DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1355bd78-3c21-4bf1-a585-8e66bf03cbb1/volumes/kubernetes.io~projected/kube-api-access-pwz8n DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32707107cd00f64a2476a6dec103b0344f1030b41b0215609c210c8b9df3e722/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2213d2f25e56bd046552d5fbeee6578e972ceee79105ea992c5f39fc597f7a4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0a8c414d39208f5de53d130c60b57a8853d4a49b3f488129d0a0a8cf74104bd/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a6f16d7-d3db-4415-bf36-c05b4200e860/volumes/kubernetes.io~projected/kube-api-access-c4fkp DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04c2c55c-c14f-438b-b025-0ccf42d1d253/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eec9d2b-a414-4463-9745-5a9dc01f4061/volumes/kubernetes.io~projected/kube-api-access-fskw4 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9b8922c7067586fb8bb18e490ed8c7840d5a3b0fe7edd4ebd25815c52f795b4/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f5c3af47d18bec6d547761a3c188a17ae938b486b75f41386bfd708691d8871/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/e7044cbc-8ac5-4ada-a632-8011de51e2d9/volumes/kubernetes.io~projected/kube-api-access-4vpp4 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b28bf8b24c8b422a759b460e7f8389d07d5ae753a709a39ff75cf4d4b913e63/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5448759-b9dd-422e-92dc-e87aedb5ef98/volumes/kubernetes.io~projected/kube-api-access-vcgcw DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa25a759c0412db8e72b10dafe7b9e4057b1199d2f7e6ab1660819c064f38ade/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/858ded61c43a4a284c1b1fa9702be95339bc49ff36badf9a81a08b355b37e266/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb6e4ae6383da293292d604f9068731dc24805129f0e6862551a9b2ff6762bf/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26cefc69324ec4e2912828e4cecd46f407dcee0bc32cc4c5d2da40c89ee9b603/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d/volumes/kubernetes.io~projected/kube-api-access-24xfn DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85ae7da1e6298a66873ef2bbeb906ebe7079f91b514660b053cc77834135d355/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee642aa1-5652-4b69-acd8-0fb5c48cd430/volumes/kubernetes.io~projected/kube-api-access-hzxmf DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f/volumes/kubernetes.io~projected/kube-api-access-w47gd DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b28bf8b24c8b42 MacAddress:32:66:6c:fb:ec:f8 Speed:10000 Mtu:8901} {Name:10a3c69b21e3fa4 MacAddress:3e:09:5c:7b:9a:ea Speed:10000 Mtu:8901} {Name:858ded61c43a4a2 MacAddress:76:1b:19:25:73:be Speed:10000 Mtu:8901} {Name:9f5c3af47d18bec MacAddress:66:b4:ea:02:3e:38 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f5:d5:ed:2f Speed:0 Mtu:9001} {Name:br-int MacAddress:46:08:23:66:f1:f1 Speed:0 Mtu:8901} {Name:cdf32038a377767 MacAddress:2e:c0:21:e1:41:5a Speed:10000 Mtu:8901} {Name:e8cff32d2a95785 MacAddress:46:e4:56:ec:76:cf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f5:d5:ed:2f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a2:81:b8:fd:27:e7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:52:03:51:55:dc:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.853425    4531 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.853493    4531 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.853801    4531 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.853822    4531 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-112.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.854156    4531 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.854186    4531 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.854200    4531 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.854243    4531 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.854358    4531 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855178    4531 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855292    4531 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855328    4531 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855341    4531 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855350    4531 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855359    4531 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:40:54.855990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.855367    4531 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:40:54.856686 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.856125    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:40:54.856686 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.856137    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:40:54.856686 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.856634    4531 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:40:54.858612 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858599    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:40:54.858612 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858612    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:40:54.858709 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858699    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 00:40:54.858889 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858878    4531 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:40:54.858924 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858907    4531 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:40:54.858999 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.858990    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859004    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859011    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859017    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859022    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859028    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859033    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859039    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:40:54.859042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859045    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:40:54.859359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859051    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:40:54.859359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859067    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:40:54.859359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859076    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:40:54.859359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859102    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:40:54.859359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.859108    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:40:54.862387 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.862373    4531 server.go:1263] "Started kubelet"
Nov 03 00:40:54.862482 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.862453    4531 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:40:54.862816 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.862757    4531 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:40:54.862926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.862833    4531 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:40:54.862877 ip-10-0-136-112 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:40:54.863435 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.863152    4531 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:40:54.863700 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.863681    4531 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867025    4531 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867453    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867513    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 00:35:16 +0000 UTC" deadline="2028-07-23 03:21:07.895801858 +0000 UTC"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867541    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15074h40m13.028263325s"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867604    4531 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867618    4531 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.867621    4531 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:40:54.868324 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:54.867607    4531 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-112.us-east-2.compute.internal\" not found"
Nov 03 00:40:54.869773 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.869752    4531 factory.go:55] Registering systemd factory
Nov 03 00:40:54.869773 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.869775    4531 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:40:54.870178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870125    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1e3ecf-7531-454d-93a5-5a6cce661c27" volumeName="kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume" seLinuxMountContext=""
Nov 03 00:40:54.870228 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870181    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7" volumeName="kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr" seLinuxMountContext=""
Nov 03 00:40:54.870228 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870197    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91635f66-bcd2-4510-ad4b-d0412ace3255" volumeName="kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d" seLinuxMountContext=""
Nov 03 00:40:54.870228 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870215    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d495ba-cbe7-4415-b87d-968ead7c09a3" volumeName="kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870228    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7044cbc-8ac5-4ada-a632-8011de51e2d9" volumeName="kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870244    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7044cbc-8ac5-4ada-a632-8011de51e2d9" volumeName="kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870257    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfad136-a64a-4525-ae3a-d77fb5af1084" volumeName="kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870270    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eec9d2b-a414-4463-9745-5a9dc01f4061" volumeName="kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870290    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04c2c55c-c14f-438b-b025-0ccf42d1d253" volumeName="kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870316    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ebb569-3603-406a-9999-42f66e578212" volumeName="kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle" seLinuxMountContext=""
Nov 03 00:40:54.870345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870339    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfad136-a64a-4525-ae3a-d77fb5af1084" volumeName="kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870357    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee31d79-9466-44c0-93e5-b69d44ebaf0b" volumeName="kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870381    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6f16d7-d3db-4415-bf36-c05b4200e860" volumeName="kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870399    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6f16d7-d3db-4415-bf36-c05b4200e860" volumeName="kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870430    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91635f66-bcd2-4510-ad4b-d0412ace3255" volumeName="kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870447    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5448759-b9dd-422e-92dc-e87aedb5ef98" volumeName="kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870487    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1355bd78-3c21-4bf1-a585-8e66bf03cbb1" volumeName="kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870511    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1355bd78-3c21-4bf1-a585-8e66bf03cbb1" volumeName="kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n" seLinuxMountContext=""
Nov 03 00:40:54.870536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870529    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c45381d-5957-4d81-847f-5e3ad1b1780c" volumeName="kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870552    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6f16d7-d3db-4415-bf36-c05b4200e860" volumeName="kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870573    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eec9d2b-a414-4463-9745-5a9dc01f4061" volumeName="kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870591    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8d638b-12ea-4bf4-b6a2-ec9b4107876f" volumeName="kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870614    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5448759-b9dd-422e-92dc-e87aedb5ef98" volumeName="kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870632    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5448759-b9dd-422e-92dc-e87aedb5ef98" volumeName="kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870655    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ebb569-3603-406a-9999-42f66e578212" volumeName="kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870673    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c45381d-5957-4d81-847f-5e3ad1b1780c" volumeName="kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870692    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8d638b-12ea-4bf4-b6a2-ec9b4107876f" volumeName="kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870715    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f8639c-a7fe-4b29-ba5a-8f865346ae9d" volumeName="kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870746    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee642aa1-5652-4b69-acd8-0fb5c48cd430" volumeName="kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf" seLinuxMountContext=""
Nov 03 00:40:54.870765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870763    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" volumeName="kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870779    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" volumeName="kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870791    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" volumeName="kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870805    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7044cbc-8ac5-4ada-a632-8011de51e2d9" volumeName="kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870823    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1e3ecf-7531-454d-93a5-5a6cce661c27" volumeName="kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870837    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc1e3ecf-7531-454d-93a5-5a6cce661c27" volumeName="kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870854    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ebb569-3603-406a-9999-42f66e578212" volumeName="kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870868    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7044cbc-8ac5-4ada-a632-8011de51e2d9" volumeName="kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870880    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27ebb569-3603-406a-9999-42f66e578212" volumeName="kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870898    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7" volumeName="kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870910    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eec9d2b-a414-4463-9745-5a9dc01f4061" volumeName="kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870927    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" volumeName="kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870940    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8d638b-12ea-4bf4-b6a2-ec9b4107876f" volumeName="kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870953    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7044cbc-8ac5-4ada-a632-8011de51e2d9" volumeName="kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870965    4531 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870972    4531 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:40:54.871112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.870985    4531 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871147    4531 factory.go:153] Registering CRI-O factory
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871175    4531 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871232    4531 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871250    4531 factory.go:103] Registering Raw factory
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871262    4531 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:40:54.872651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.871558    4531 manager.go:319] Starting recovery of all containers
Nov 03 00:40:54.882187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.876748    4531 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:54.882187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.877094    4531 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 00:40:54.882187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.877531    4531 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:54.893372 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.893306    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:40:54.894769 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.894740    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:40:54.894846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.894772    4531 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:40:54.895110 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.895090    4531 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:40:54.895190 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:54.895146    4531 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 00:40:54.898650 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.898571    4531 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:40:54.906808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906780    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="76d95a76252b845e4908e57e5705f694ba0a599d14c249659437f2917a1b2022" exitCode=0
Nov 03 00:40:54.906883 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906811    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="eecf213bd7b63fa7aa991d39fbf37bb5a97b360fbd5e78208941358c85fa386e" exitCode=0
Nov 03 00:40:54.906883 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906822    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="50443b8360a359c88ba28256d2b13d3a5388b0d4a5641fc7b0ee4db2abc22a51" exitCode=0
Nov 03 00:40:54.906883 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906831    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="54280ec88b5cb097eb3a75eb2b47e4607a9df72a5918f69488d0feea05a5fc7c" exitCode=0
Nov 03 00:40:54.906883 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906841    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="52be3ea438888548f08c53c62674ca67513bb18b730b8af11159f0d0e2d03487" exitCode=0
Nov 03 00:40:54.906883 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.906853    4531 generic.go:356] "Generic (PLEG): container finished" podID="a548ec83-53fc-4a5c-a5d8-574b6ee5e971" containerID="b33e6070a90738bb8cba6663447e7c3fca8be0cad9124cdb673b4b3e9a397858" exitCode=0
Nov 03 00:40:54.925554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.925530    4531 generic.go:356] "Generic (PLEG): container finished" podID="4bc7fd51fa7784f0272b9c9a23f1eea8" containerID="61910b81cdc69058c6e5b6b1a4ce964ebf43bbe0a941436fc5e001ab54b16c42" exitCode=0
Nov 03 00:40:54.944025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.944004    4531 manager.go:324] Recovery completed
Nov 03 00:40:54.958075 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958057    4531 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:40:54.958075 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958071    4531 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:40:54.958224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958087    4531 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:40:54.958224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958210    4531 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958221    4531 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958238    4531 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958244    4531 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958249    4531 policy_none.go:50] "Start"
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958257    4531 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:40:54.958285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958266    4531 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:40:54.958430 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958386    4531 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 00:40:54.958430 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958392    4531 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 00:40:54.958430 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.958398    4531 policy_none.go:44] "Start"
Nov 03 00:40:54.962515 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.962484    4531 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:40:54.965201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.965186    4531 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:40:54.965870 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:54.965214    4531 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:40:54.965870 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.965870    4531 server.go:87] "Starting device plugin registration server"
Nov 03 00:40:54.966117 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966106    4531 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:40:54.966178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966117    4531 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:40:54.966268 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966255    4531 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:40:54.966385 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966327    4531 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:40:54.966385 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966371    4531 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:40:54.966385 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966377    4531 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:40:54.966527 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.966424    4531 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:40:54.966361742Z","UUID":"6a0d83d1-d127-4cb8-90ab-351bce189b36","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:40:54.967603 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:54.967582    4531 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:40:54.968037 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.968018    4531 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:40:54.968128 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.968044    4531 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:40:54.995850 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.995775    4531 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"]
Nov 03 00:40:54.996289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996254    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4ecdbd9fc26c68793bebc9a7943385b7","Type":"ContainerStarted","Data":"26d7f07a20f2f45797335998eaa7188a21dbed31b44914427380b736097f0c0f"}
Nov 03 00:40:54.996393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996295    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4ecdbd9fc26c68793bebc9a7943385b7","Type":"ContainerStarted","Data":"4b1f9cda2cda156a45a5566348648d0c44410f1a856bc9cd2dbe7f2f3d1c8ed3"}
Nov 03 00:40:54.996393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996330    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:54.996393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996354    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerStarted","Data":"3b01ab80c9438f43e30341d35b3f71d6118062110380502a37805e698cf3a51d"}
Nov 03 00:40:54.996393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerDied","Data":"61910b81cdc69058c6e5b6b1a4ce964ebf43bbe0a941436fc5e001ab54b16c42"}
Nov 03 00:40:54.996546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996394    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal" event={"ID":"4bc7fd51fa7784f0272b9c9a23f1eea8","Type":"ContainerStarted","Data":"f0fbf4ab5a610aab1badde887523c41432b7ac07e7fca5b45747238e72c09f77"}
Nov 03 00:40:54.996546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:54.996458    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.008742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.008727    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:55.008803 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:55.008770    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.009629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.009619    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:55.009662 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:55.009643    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.066809 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.066782    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:40:55.068060 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.068042    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 00:40:55.068153 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.068071    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 00:40:55.068153 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.068081    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-112.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 00:40:55.068153 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.068120    4531 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.077254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.077234    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.077355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.077258    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.077355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.077276    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.082398 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.082382    4531 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.082474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.082430    4531 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178327 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178290    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178327 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178239    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4ecdbd9fc26c68793bebc9a7943385b7-config\") pod \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4ecdbd9fc26c68793bebc9a7943385b7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178355    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178372    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178407    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.178554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.178436    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4bc7fd51fa7784f0272b9c9a23f1eea8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" (UID: \"4bc7fd51fa7784f0272b9c9a23f1eea8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.856401 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.856359    4531 apiserver.go:52] "Watching apiserver"
Nov 03 00:40:55.865900 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.865876    4531 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:40:55.867259 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.867225    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-5ff5v","openshift-multus/multus-5j8gb","kube-system/konnectivity-agent-cbkz7","openshift-dns/node-resolver-wrzvg","openshift-image-registry/node-ca-l9wpw","openshift-multus/network-metrics-daemon-rlbmn","openshift-service-ca/service-ca-5fffc97bdf-xvtfb","openshift-network-operator/iptables-alerter-m9fbf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs","kube-system/global-pull-secret-syncer-7vg7f","kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-cw99z","openshift-insights/insights-runtime-extractor-8x65q","openshift-network-diagnostics/network-check-target-9g7gh","openshift-insights/periodic-gathering-wj6b6-dbm2v","openshift-ingress-canary/ingress-canary-hxbss","openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-7fbfg","openshift-dns/dns-default-dcrmw"]
Nov 03 00:40:55.870499 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.870473    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:55.873224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.873202    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.873784 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.873767    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.877978 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.877960    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.878363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.878346    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:40:55.878453 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.878439    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.878527 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.878499    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:40:55.878592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.878581    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.879081 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879066    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.879355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879339    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:40:55.879418 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879355    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.879463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879417    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:40:55.879510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879476    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.879549 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879504    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.879902 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879885    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.879956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879919    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:40:55.879994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879951    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.879994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879969    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880079 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.879995    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:40:55.880079 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880053    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880151    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:40:55.880187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880178    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880182    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:40:55.880275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880235    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880295    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880331    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880349    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880355    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880365    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880360    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880359    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880400    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880412    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.880873 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880499    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:40:55.880873 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.880545    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:40:55.881054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881041    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:40:55.881224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881270 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881243    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881270 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881263    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881365 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881379    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881456 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881441    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881506 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881488    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881549 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881523    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881549 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881540    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881639 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881639 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881582    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881639 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881639 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881655    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskw4\" (UniqueName: \"kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881681    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-daemon-config\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881749    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881816    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:55.881859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881848    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7eec9d2b-a414-4463-9745-5a9dc01f4061-cni-binary-copy\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.882178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.881968    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.882178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882014    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 00:40:55.882178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882082    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9v27r\""
Nov 03 00:40:55.882178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882088    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 00:40:55.882368 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882211    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 00:40:55.882368 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882352    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:40:55.882737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882715    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:40:55.882808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882758    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:40:55.883220 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.882758    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:40:55.883306 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.883232    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28tgg\""
Nov 03 00:40:55.883306 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.883300    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:40:55.883577 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.883557    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d54vs\""
Nov 03 00:40:55.883796 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.883591    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-97hfk\""
Nov 03 00:40:55.883972 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.883922    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:40:55.884028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884008    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k7bjp\""
Nov 03 00:40:55.884074 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884047    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tq982\""
Nov 03 00:40:55.884463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884440    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tg82s\""
Nov 03 00:40:55.884538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884487    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dx4w2\""
Nov 03 00:40:55.884538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884514    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4zb6t\""
Nov 03 00:40:55.884633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884554    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9l2l\""
Nov 03 00:40:55.884633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884572    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-42dxl\""
Nov 03 00:40:55.884633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884588    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:40:55.884633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884609    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:40:55.884960 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884922    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-229lw\""
Nov 03 00:40:55.885062 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884981    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:40:55.885062 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.884933    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz52v\""
Nov 03 00:40:55.885062 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885055    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 00:40:55.885230 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885179    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4c8qt\""
Nov 03 00:40:55.885277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885258    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mmgk\""
Nov 03 00:40:55.885277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885267    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vmfsq\""
Nov 03 00:40:55.885367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885311    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7trdq\""
Nov 03 00:40:55.885470 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885455    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:40:55.885615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.885589    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kp9k4\""
Nov 03 00:40:55.889100 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.889083    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskw4\" (UniqueName: \"kubernetes.io/projected/7eec9d2b-a414-4463-9745-5a9dc01f4061-kube-api-access-fskw4\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.889283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.889268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn7s\" (UniqueName: \"kubernetes.io/projected/2ee31d79-9466-44c0-93e5-b69d44ebaf0b-kube-api-access-6kn7s\") pod \"network-check-target-9g7gh\" (UID: \"2ee31d79-9466-44c0-93e5-b69d44ebaf0b\") " pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:55.933741 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.933717    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.933861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.933803    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.949018 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.948996    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:55.949090 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:55.949031    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.952790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.952773    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:40:55.952913 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:55.952813    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-112.us-east-2.compute.internal"
Nov 03 00:40:55.969050 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.969025    4531 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:40:55.982231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982202    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982302    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982328    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982333    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-cnibin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982351    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982375    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982416    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.982471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xfn\" (UniqueName: \"kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982476    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-config\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982480    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-multus\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982509    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-hostroot\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6f16d7-d3db-4415-bf36-c05b4200e860-tmp-dir\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982574    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-env-overrides\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982605    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.982691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982658    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq85c\" (UniqueName: \"kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982808    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-k8s-cni-cncf-io\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982815    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982812    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982857    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982874    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.982908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982893    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.983088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.983088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:55.983088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.982956    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:55.983088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47gd\" (UniqueName: \"kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:55.983088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983051    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1e3ecf-7531-454d-93a5-5a6cce661c27-data-volume\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.983275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983097    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-binary-copy\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.983275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983135    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:55.983275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-etc-kubernetes\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.983275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-key\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:55.983275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxmf\" (UniqueName: \"kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf\") pod \"migrator-66df5d496b-st6fr\" (UID: \"ee642aa1-5652-4b69-acd8-0fb5c48cd430\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:55.983409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983289    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.983409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.983409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983343    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.983409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983398    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983425    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c45381d-5957-4d81-847f-5e3ad1b1780c-agent-certs\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983444    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-tuned\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983478    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.983551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983524    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-socket-dir-parent\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.983723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983553    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:55.983723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983593    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:55.983723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bfad136-a64a-4525-ae3a-d77fb5af1084-tmp-dir\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:55.983723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983654    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.983893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.983893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.983893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983812    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:55.983893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983839    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91635f66-bcd2-4510-ad4b-d0412ace3255-iptables-alerter-script\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:55.983893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983847    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:55.984116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.983916    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.984116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.984116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984057    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-signing-cabundle\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:55.984116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984062    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-cert\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:55.984116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984095    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6f16d7-d3db-4415-bf36-c05b4200e860-config-volume\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984208    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-multus-certs\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984194    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984237    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn8d\" (UniqueName: \"kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984254    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984267    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984287    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984305    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-conf-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984288    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-run-netns\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984361    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-system-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984365    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984387    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984402    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984418    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984448    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5972d\" (UniqueName: \"kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c45381d-5957-4d81-847f-5e3ad1b1780c-konnectivity-ca\") pod \"konnectivity-agent-cbkz7\" (UID: \"3c45381d-5957-4d81-847f-5e3ad1b1780c\") " pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984521    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984574    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovn-node-metrics-cert\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984613    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984636    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984665    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.984726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984702    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984785    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984815    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5448759-b9dd-422e-92dc-e87aedb5ef98-tmp\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984833    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984903    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cjr\" (UniqueName: \"kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984933    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984967    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.984986    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985021    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-cni-bin\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985078    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fkp\" (UniqueName: \"kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985105    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgcw\" (UniqueName: \"kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985182    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985210    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985227    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985259    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985364    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9m6\" (UniqueName: \"kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985405    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985431    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7044cbc-8ac5-4ada-a632-8011de51e2d9-ovnkube-script-lib\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985488    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpp4\" (UniqueName: \"kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985560    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985571    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985589    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.985615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985613    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985635    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985644    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985651    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985676    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985711    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985861    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-multus-cni-dir\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985891    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjp97\" (UniqueName: \"kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985909    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985925    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985928    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985939    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz8n\" (UniqueName: \"kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.985999    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:55.986028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.986031    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.986486 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.986056    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.986486 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.986119    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eec9d2b-a414-4463-9745-5a9dc01f4061-host-var-lib-kubelet\") pod \"multus-5j8gb\" (UID: \"7eec9d2b-a414-4463-9745-5a9dc01f4061\") " pod="openshift-multus/multus-5j8gb"
Nov 03 00:40:55.986486 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.986230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-serviceca\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:55.986486 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.986332    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04c2c55c-c14f-438b-b025-0ccf42d1d253-original-pull-secret\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:55.994011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.993991    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-st6fr_ee642aa1-5652-4b69-acd8-0fb5c48cd430/migrator/0.log"
Nov 03 00:40:55.994866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.994850    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn8d\" (UniqueName: \"kubernetes.io/projected/2bfad136-a64a-4525-ae3a-d77fb5af1084-kube-api-access-wrn8d\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:55.996443 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.996427    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"periodic-gathering-wj6b6-dbm2v\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") " pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:40:55.996633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.996618    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9m6\" (UniqueName: \"kubernetes.io/projected/b7d495ba-cbe7-4415-b87d-968ead7c09a3-kube-api-access-bj9m6\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:55.996682 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.996669    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fkp\" (UniqueName: \"kubernetes.io/projected/6a6f16d7-d3db-4415-bf36-c05b4200e860-kube-api-access-c4fkp\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:55.998917 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.998899    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgcw\" (UniqueName: \"kubernetes.io/projected/c5448759-b9dd-422e-92dc-e87aedb5ef98-kube-api-access-vcgcw\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:55.999122 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.999102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz8n\" (UniqueName: \"kubernetes.io/projected/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-kube-api-access-pwz8n\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:55.999204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.999154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cjr\" (UniqueName: \"kubernetes.io/projected/43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7-kube-api-access-c2cjr\") pod \"ingress-canary-hxbss\" (UID: \"43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7\") " pod="openshift-ingress-canary/ingress-canary-hxbss"
Nov 03 00:40:55.999276 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.999258    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxmf\" (UniqueName: \"kubernetes.io/projected/ee642aa1-5652-4b69-acd8-0fb5c48cd430-kube-api-access-hzxmf\") pod \"migrator-66df5d496b-st6fr\" (UID: \"ee642aa1-5652-4b69-acd8-0fb5c48cd430\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-st6fr"
Nov 03 00:40:55.999566 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:55.999551    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjp97\" (UniqueName: \"kubernetes.io/projected/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-kube-api-access-hjp97\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.000255 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.000236    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq85c\" (UniqueName: \"kubernetes.io/projected/fc1e3ecf-7531-454d-93a5-5a6cce661c27-kube-api-access-wq85c\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:56.000910 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.000888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpp4\" (UniqueName: \"kubernetes.io/projected/e7044cbc-8ac5-4ada-a632-8011de51e2d9-kube-api-access-4vpp4\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.000982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.000936    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xfn\" (UniqueName: \"kubernetes.io/projected/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kube-api-access-24xfn\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.000982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.000933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47gd\" (UniqueName: \"kubernetes.io/projected/bf8d638b-12ea-4bf4-b6a2-ec9b4107876f-kube-api-access-w47gd\") pod \"service-ca-5fffc97bdf-xvtfb\" (UID: \"bf8d638b-12ea-4bf4-b6a2-ec9b4107876f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xvtfb"
Nov 03 00:40:56.001260 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.001244    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5972d\" (UniqueName: \"kubernetes.io/projected/91635f66-bcd2-4510-ad4b-d0412ace3255-kube-api-access-5972d\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:56.087048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:56.087048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087048    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087064    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:56.087248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91635f66-bcd2-4510-ad4b-d0412ace3255-host-slash\") pod \"iptables-alerter-m9fbf\" (UID: \"91635f66-bcd2-4510-ad4b-d0412ace3255\") " pod="openshift-network-operator/iptables-alerter-m9fbf"
Nov 03 00:40:56.087248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087187    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-netd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087283    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087298    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087314    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087335    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.087359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087350    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087365    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087413    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-sys-fs\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-node-log\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-conf\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087415    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087476    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087482    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-host\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-socket-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087448    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-cni-bin\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-dbus\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087499    4531 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087535    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1355bd78-3c21-4bf1-a585-8e66bf03cbb1-host\") pod \"node-ca-l9wpw\" (UID: \"1355bd78-3c21-4bf1-a585-8e66bf03cbb1\") " pod="openshift-image-registry/node-ca-l9wpw"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087495    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-log-socket\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087596    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087631    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087644    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087659    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087689    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087695    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysconfig\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087710    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087741    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-systemd\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087747    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1e3ecf-7531-454d-93a5-5a6cce661c27-crio-socket\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087767    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-lib-modules\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087768    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087795    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-kubernetes\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:56.087799    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:56.087886    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:56.587871091 +0000 UTC m=+1.801614766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087917    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087934    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-systemd-units\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-var-lib-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087969    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-cnibin\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087975    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.087986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088011    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088037    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088073    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-sysctl-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088088    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-etc-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088126    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-slash\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088119    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088126    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088182    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-var-lib-kubelet\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088247    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088261    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-openvswitch\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088247    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-registration-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088282    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04c2c55c-c14f-438b-b025-0ccf42d1d253-kubelet-config\") pod \"global-pull-secret-syncer-7vg7f\" (UID: \"04c2c55c-c14f-438b-b025-0ccf42d1d253\") " pod="kube-system/global-pull-secret-syncer-7vg7f"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088272    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088300    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a548ec83-53fc-4a5c-a5d8-574b6ee5e971-system-cni-dir\") pod \"multus-additional-cni-plugins-cw99z\" (UID: \"a548ec83-53fc-4a5c-a5d8-574b6ee5e971\") " pod="openshift-multus/multus-additional-cni-plugins-cw99z"
Nov 03 00:40:56.088296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088314    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088349    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088365    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088379    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088385    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-run-ovn\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-modprobe-d\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088399    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088409    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-run\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088430    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088444    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-sys\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-kubelet\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088485    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5448759-b9dd-422e-92dc-e87aedb5ef98-etc-systemd\") pod \"tuned-7fbfg\" (UID: \"c5448759-b9dd-422e-92dc-e87aedb5ef98\") " pod="openshift-cluster-node-tuning-operator/tuned-7fbfg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088512    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7044cbc-8ac5-4ada-a632-8011de51e2d9-host-run-netns\") pod \"ovnkube-node-5ff5v\" (UID: \"e7044cbc-8ac5-4ada-a632-8011de51e2d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088513    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6f8639c-a7fe-4b29-ba5a-8f865346ae9d-device-dir\") pod \"aws-ebs-csi-driver-node-7s8hs\" (UID: \"c6f8639c-a7fe-4b29-ba5a-8f865346ae9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7s8hs"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.088558    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2bfad136-a64a-4525-ae3a-d77fb5af1084-hosts-file\") pod \"node-resolver-wrzvg\" (UID: \"2bfad136-a64a-4525-ae3a-d77fb5af1084\") " pod="openshift-dns/node-resolver-wrzvg"
Nov 03 00:40:56.090303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.089736    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7d495ba-cbe7-4415-b87d-968ead7c09a3-metrics-certs\") pod \"network-metrics-daemon-rlbmn\" (UID: \"b7d495ba-cbe7-4415-b87d-968ead7c09a3\") " pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:56.090803 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.090440    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6f16d7-d3db-4415-bf36-c05b4200e860-metrics-tls\") pod \"dns-default-dcrmw\" (UID: \"6a6f16d7-d3db-4415-bf36-c05b4200e860\") " pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:56.173600 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.173516    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbmn"
Nov 03 00:40:56.173749 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.173713    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:56.486789 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.486711    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlbmn"]
Nov 03 00:40:56.490923 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:56.490887    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d495ba_cbe7_4415_b87d_968ead7c09a3.slice/crio-89373f6a4186ab828eada60c3d1c36893e82d8e0f16fbda1e3ea59282dca55bc WatchSource:0}: Error finding container 89373f6a4186ab828eada60c3d1c36893e82d8e0f16fbda1e3ea59282dca55bc: Status 404 returned error can't find the container with id 89373f6a4186ab828eada60c3d1c36893e82d8e0f16fbda1e3ea59282dca55bc
Nov 03 00:40:56.492897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.492880    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:40:56.510962 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.510936    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dcrmw"]
Nov 03 00:40:56.514133 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:40:56.514111    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6f16d7_d3db_4415_bf36_c05b4200e860.slice/crio-4651b018f911739c80772ae0019002a07e52e658b77f47150d25fd281ad496bc WatchSource:0}: Error finding container 4651b018f911739c80772ae0019002a07e52e658b77f47150d25fd281ad496bc: Status 404 returned error can't find the container with id 4651b018f911739c80772ae0019002a07e52e658b77f47150d25fd281ad496bc
Nov 03 00:40:56.592842 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.592806    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:56.592978 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:56.592961    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:56.593037 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:56.593027    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:57.593012147 +0000 UTC m=+2.806755798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:56.937214 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.937178    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbmn" event={"ID":"b7d495ba-cbe7-4415-b87d-968ead7c09a3","Type":"ContainerStarted","Data":"89373f6a4186ab828eada60c3d1c36893e82d8e0f16fbda1e3ea59282dca55bc"}
Nov 03 00:40:56.938269 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:56.938234    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcrmw" event={"ID":"6a6f16d7-d3db-4415-bf36-c05b4200e860","Type":"ContainerStarted","Data":"4651b018f911739c80772ae0019002a07e52e658b77f47150d25fd281ad496bc"}
Nov 03 00:40:57.048321 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:57.048287    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:40:57.598768 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:57.598731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:57.598956 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:57.598913    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:57.599023 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:57.599011    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:40:59.598987764 +0000 UTC m=+4.812731434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:58.947544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:58.947503    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcrmw" event={"ID":"6a6f16d7-d3db-4415-bf36-c05b4200e860","Type":"ContainerStarted","Data":"0cf0fb9fa25ff313c23211b761d2e20783e3885a31f2508f4712914b2ca9992e"}
Nov 03 00:40:58.948044 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:58.947597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dcrmw" event={"ID":"6a6f16d7-d3db-4415-bf36-c05b4200e860","Type":"ContainerStarted","Data":"1e4e9243bbad3e5f0b020b44ad4c6c5c8c1f2804f94c31671a008d7fe88e0e84"}
Nov 03 00:40:58.948044 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:58.947628    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:40:58.949259 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:58.949237    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbmn" event={"ID":"b7d495ba-cbe7-4415-b87d-968ead7c09a3","Type":"ContainerStarted","Data":"65323e8c87a2fa1b991129f4f5a766777dcf4989bbb0fde9c798bcf935536d73"}
Nov 03 00:40:58.949373 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:58.949276    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbmn" event={"ID":"b7d495ba-cbe7-4415-b87d-968ead7c09a3","Type":"ContainerStarted","Data":"db76ee099e9f2dd18f04702de84d780029766c6d47be98eaa9ed5693a2dc2d48"}
Nov 03 00:40:59.612595 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:40:59.612562    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:40:59.612758 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:59.612686    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:40:59.612758 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:40:59.612746    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls podName:fc1e3ecf-7531-454d-93a5-5a6cce661c27 nodeName:}" failed. No retries permitted until 2026-11-03 00:41:03.612730608 +0000 UTC m=+8.826474275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8x65q" (UID: "fc1e3ecf-7531-454d-93a5-5a6cce661c27") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:41:01.154894 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:01.154860    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:41:01.547300 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:01.547234    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:41:01.929250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:01.929225    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5ff5v"
Nov 03 00:41:03.639143 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:03.639105    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:41:03.641439 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:03.641417    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1e3ecf-7531-454d-93a5-5a6cce661c27-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8x65q\" (UID: \"fc1e3ecf-7531-454d-93a5-5a6cce661c27\") " pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:41:03.674418 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:03.674386    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8x65q"
Nov 03 00:41:03.951179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:03.951037    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8x65q"]
Nov 03 00:41:03.955179 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:03.955136    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1e3ecf_7531_454d_93a5_5a6cce661c27.slice/crio-3da604147087dce45fff5271d0e16529bf9c4109aa04b5594815281cbd70864e WatchSource:0}: Error finding container 3da604147087dce45fff5271d0e16529bf9c4109aa04b5594815281cbd70864e: Status 404 returned error can't find the container with id 3da604147087dce45fff5271d0e16529bf9c4109aa04b5594815281cbd70864e
Nov 03 00:41:03.964990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:03.964964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x65q" event={"ID":"fc1e3ecf-7531-454d-93a5-5a6cce661c27","Type":"ContainerStarted","Data":"3da604147087dce45fff5271d0e16529bf9c4109aa04b5594815281cbd70864e"}
Nov 03 00:41:04.972389 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:04.972309    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x65q" event={"ID":"fc1e3ecf-7531-454d-93a5-5a6cce661c27","Type":"ContainerStarted","Data":"7ea2e8f03af707cb9927c97cac239d5207facbbe47e9eae99d3c5e7eb5d3cc6f"}
Nov 03 00:41:04.972389 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:04.972362    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x65q" event={"ID":"fc1e3ecf-7531-454d-93a5-5a6cce661c27","Type":"ContainerStarted","Data":"59b45f4d302f131052e1f4633a739b65a79bd8b7183d4ae98efc5928d1fec53e"}
Nov 03 00:41:06.980204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:06.980144    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8x65q" event={"ID":"fc1e3ecf-7531-454d-93a5-5a6cce661c27","Type":"ContainerStarted","Data":"6bc817940cb0f477c65ab8591f2f0d9b36ca9efa18dbe650af7f533c0e7034b0"}
Nov 03 00:41:08.954980 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:08.954940    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dcrmw"
Nov 03 00:41:09.847710 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.847674    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:41:09.851179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.851139    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:09.864276 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.864248    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:41:09.882753 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:41:09.882717    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:41:09.982274 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.982242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flghj\" (UniqueName: \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:09.982703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.982280    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:09.982703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.982299    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:09.982703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:09.982364    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.083423 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.083385    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flghj\" (UniqueName: \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.083615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.083444    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.083615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.083469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.083615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.083495    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.083948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.083924    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.084365 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.084341    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.084603 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.084586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.102435 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.102386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flghj\" (UniqueName: \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\") pod \"periodic-gathering-dftf6-zs2wg\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") " pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.163745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.163712    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:41:10.433878 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.433805    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:41:10.437326 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:10.437297    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fea578_3e8d_45b3_966a_af5c5a08d0da.slice/crio-e96341f37270457b78e42381264c5a081875d4ebed730b9b59db9f75bae4785e WatchSource:0}: Error finding container e96341f37270457b78e42381264c5a081875d4ebed730b9b59db9f75bae4785e: Status 404 returned error can't find the container with id e96341f37270457b78e42381264c5a081875d4ebed730b9b59db9f75bae4785e
Nov 03 00:41:10.994547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.994510    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" event={"ID":"04fea578-3e8d-45b3-966a-af5c5a08d0da","Type":"ContainerStarted","Data":"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"}
Nov 03 00:41:10.995048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:10.994559    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" event={"ID":"04fea578-3e8d-45b3-966a-af5c5a08d0da","Type":"ContainerStarted","Data":"e96341f37270457b78e42381264c5a081875d4ebed730b9b59db9f75bae4785e"}
Nov 03 00:41:11.019839 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:11.019788    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" podStartSLOduration=2.01977351 podStartE2EDuration="2.01977351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:11.018268994 +0000 UTC m=+16.232012665" watchObservedRunningTime="2026-11-03 00:41:11.01977351 +0000 UTC m=+16.233517178"
Nov 03 00:41:11.653507 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:11.653478    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcrmw_6a6f16d7-d3db-4415-bf36-c05b4200e860/dns/0.log"
Nov 03 00:41:11.852979 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:11.852952    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcrmw_6a6f16d7-d3db-4415-bf36-c05b4200e860/kube-rbac-proxy/0.log"
Nov 03 00:41:12.856203 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:12.856150    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wrzvg_2bfad136-a64a-4525-ae3a-d77fb5af1084/dns-node-resolver/0.log"
Nov 03 00:41:13.854781 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:13.854756    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9wpw_1355bd78-3c21-4bf1-a585-8e66bf03cbb1/node-ca/0.log"
Nov 03 00:41:14.453846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:14.453821    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hxbss_43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/serve-healthcheck-canary/0.log"
Nov 03 00:41:17.284194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.284137    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6mk4l"]
Nov 03 00:41:17.287790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.287768    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.301466 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.301448    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxq4b\""
Nov 03 00:41:17.301652 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.301636    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 00:41:17.302730 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.302716    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:41:17.302800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.302755    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:41:17.302996 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.302982    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 00:41:17.303060 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.303035    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 00:41:17.303100 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.303067    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 00:41:17.316848 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.316822    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"]
Nov 03 00:41:17.319782 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.319765    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.325968 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.325948    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-textfile\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326066 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.325975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-sys\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326066 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-tls\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326094    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-root\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326113    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326136    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-metrics-client-ca\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-wtmp\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.326359 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.326312    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grmr\" (UniqueName: \"kubernetes.io/projected/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-kube-api-access-9grmr\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.336345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.336322    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xvb7r\""
Nov 03 00:41:17.336605 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.336592    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 00:41:17.336605 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.336598    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 00:41:17.336828 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.336815    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 00:41:17.365701 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.365667    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"]
Nov 03 00:41:17.427221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427190    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.427221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427231    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-wtmp\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427460 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427284    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grmr\" (UniqueName: \"kubernetes.io/projected/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-kube-api-access-9grmr\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427460 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427314    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-textfile\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427460 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427380    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-wtmp\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427460 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427407    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.427632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427480    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c948d3d-b8cd-43c5-b111-dba302be2a5b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.427632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-sys\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwkc\" (UniqueName: \"kubernetes.io/projected/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-api-access-xkwkc\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.427632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-tls\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427637    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-sys\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427640    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-textfile\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427661    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-root\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427694    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427738    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-root\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.427783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.428048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427792    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.428048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.427817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-metrics-client-ca\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.428350 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.428321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-metrics-client-ca\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.428350 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.428345    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.430040 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.430010    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.430040 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.430030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-node-exporter-tls\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.437151 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.437131    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grmr\" (UniqueName: \"kubernetes.io/projected/da9d2dac-e9f8-436e-8ded-c7a33eeb380a-kube-api-access-9grmr\") pod \"node-exporter-6mk4l\" (UID: \"da9d2dac-e9f8-436e-8ded-c7a33eeb380a\") " pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.528455 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528490    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c948d3d-b8cd-43c5-b111-dba302be2a5b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528511    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwkc\" (UniqueName: \"kubernetes.io/projected/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-api-access-xkwkc\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.528985 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.528963    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c948d3d-b8cd-43c5-b111-dba302be2a5b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.529289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.529261    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.529411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.529261    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.530882 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.530863    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.530964 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.530948    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.537757 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.537698    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwkc\" (UniqueName: \"kubernetes.io/projected/1c948d3d-b8cd-43c5-b111-dba302be2a5b-kube-api-access-xkwkc\") pod \"kube-state-metrics-5dc8d7d89b-6n9rm\" (UID: \"1c948d3d-b8cd-43c5-b111-dba302be2a5b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.602908 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.602877    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6mk4l"
Nov 03 00:41:17.613841 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:17.613814    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9d2dac_e9f8_436e_8ded_c7a33eeb380a.slice/crio-35fd4f6cebe7e0e450d66e7ec253f72ab895cbc32cf4e247d4b6900e31ed08f0 WatchSource:0}: Error finding container 35fd4f6cebe7e0e450d66e7ec253f72ab895cbc32cf4e247d4b6900e31ed08f0: Status 404 returned error can't find the container with id 35fd4f6cebe7e0e450d66e7ec253f72ab895cbc32cf4e247d4b6900e31ed08f0
Nov 03 00:41:17.635724 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.635699    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"
Nov 03 00:41:17.917473 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:17.917442    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm"]
Nov 03 00:41:17.920736 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:17.920709    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c948d3d_b8cd_43c5_b111_dba302be2a5b.slice/crio-14cd14afbe6d1fc61c47f8eb00abddb77556018a15b0990cada51b989bfe5ffe WatchSource:0}: Error finding container 14cd14afbe6d1fc61c47f8eb00abddb77556018a15b0990cada51b989bfe5ffe: Status 404 returned error can't find the container with id 14cd14afbe6d1fc61c47f8eb00abddb77556018a15b0990cada51b989bfe5ffe
Nov 03 00:41:18.016526 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.016489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm" event={"ID":"1c948d3d-b8cd-43c5-b111-dba302be2a5b","Type":"ContainerStarted","Data":"14cd14afbe6d1fc61c47f8eb00abddb77556018a15b0990cada51b989bfe5ffe"}
Nov 03 00:41:18.017594 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.017573    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6mk4l" event={"ID":"da9d2dac-e9f8-436e-8ded-c7a33eeb380a","Type":"ContainerStarted","Data":"35fd4f6cebe7e0e450d66e7ec253f72ab895cbc32cf4e247d4b6900e31ed08f0"}
Nov 03 00:41:18.485853 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.485815    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:41:18.489788 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.489759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.501137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501101    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501144    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501195    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501222    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501237    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501289    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501297    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501309    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 00:41:18.501360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.501324    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qswlh\""
Nov 03 00:41:18.506102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.506077    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 00:41:18.535970 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.535935    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.535972    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:41:18.536124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.535978    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlks6\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536128    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536362 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536301    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536362 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536333    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536437    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536803 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536779    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536874 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536825    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536874 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536866    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536888    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536902    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.536969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.536920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637634 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637640    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637696    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637723    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.637811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637835    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlks6\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637900    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.637980    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.638011    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.638433 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.638418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.639591    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.640106    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.640969    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.641337    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.641436    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.641483    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.641680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.641752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.641740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.642424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.642391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.642615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.642595    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.643368 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.643338    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.649044 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.648984    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlks6\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\") pod \"alertmanager-main-0\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:18.834835 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:18.834799    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:41:19.140047 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:19.140002    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:41:19.230518 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:19.230477    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f572ed_3c18_4090_a0e1_289fbdc229bf.slice/crio-71cb5cc2eecc67826c9639034aa3173f8f68bd8e41149c24d415446799d77b08 WatchSource:0}: Error finding container 71cb5cc2eecc67826c9639034aa3173f8f68bd8e41149c24d415446799d77b08: Status 404 returned error can't find the container with id 71cb5cc2eecc67826c9639034aa3173f8f68bd8e41149c24d415446799d77b08
Nov 03 00:41:20.027037 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.026949    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm" event={"ID":"1c948d3d-b8cd-43c5-b111-dba302be2a5b","Type":"ContainerStarted","Data":"b5d15ab4ace7d41fc28286378f2c0cc33b21038b9db0941eea77e263d876961d"}
Nov 03 00:41:20.027037 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.027016    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm" event={"ID":"1c948d3d-b8cd-43c5-b111-dba302be2a5b","Type":"ContainerStarted","Data":"fc5ae47adbb55fc82a6ece620ae8cab1d3510c18f96ba92097ca04f2c88d2ddf"}
Nov 03 00:41:20.027675 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.027043    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm" event={"ID":"1c948d3d-b8cd-43c5-b111-dba302be2a5b","Type":"ContainerStarted","Data":"24de8f37c06fa21fd3b8fee46ffa02557918ad185ee36affbc76d7f70e9301b7"}
Nov 03 00:41:20.029033 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.029001    4531 generic.go:356] "Generic (PLEG): container finished" podID="da9d2dac-e9f8-436e-8ded-c7a33eeb380a" containerID="d1d23db228403c36465212095ffa8a0905488d0faf2166c3204f1973fa837a0b" exitCode=0
Nov 03 00:41:20.029207 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.029092    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6mk4l" event={"ID":"da9d2dac-e9f8-436e-8ded-c7a33eeb380a","Type":"ContainerDied","Data":"d1d23db228403c36465212095ffa8a0905488d0faf2166c3204f1973fa837a0b"}
Nov 03 00:41:20.031038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.031010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"71cb5cc2eecc67826c9639034aa3173f8f68bd8e41149c24d415446799d77b08"}
Nov 03 00:41:20.096970 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:20.096923    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6n9rm" podStartSLOduration=1.477003938 podStartE2EDuration="3.096903598s" totalImagesPullingTime="1.61989966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:20.056674967 +0000 UTC m=+25.270418639" watchObservedRunningTime="2026-11-03 00:41:20.096903598 +0000 UTC m=+25.310647272"
Nov 03 00:41:21.037240 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.037132    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6mk4l" event={"ID":"da9d2dac-e9f8-436e-8ded-c7a33eeb380a","Type":"ContainerStarted","Data":"fcd2bc9714a8cede5eb70485677fb5057c47f5b67dcd879dea40d6845c3923bc"}
Nov 03 00:41:21.037742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.037265    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6mk4l" event={"ID":"da9d2dac-e9f8-436e-8ded-c7a33eeb380a","Type":"ContainerStarted","Data":"97032ffd0e87177ad99d69579a48f33728484a2b49d07b4ccc72ed52898874df"}
Nov 03 00:41:21.042523 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.042497    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68" exitCode=0
Nov 03 00:41:21.042680 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.042581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"}
Nov 03 00:41:21.063638 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.063600    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6mk4l" podStartSLOduration=2.352152629 podStartE2EDuration="4.063588485s" totalImagesPullingTime="1.636880274s" totalInitContainerRuntime="74.555582ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:21.062636897 +0000 UTC m=+26.276380570" watchObservedRunningTime="2026-11-03 00:41:21.063588485 +0000 UTC m=+26.277332207"
Nov 03 00:41:21.647283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.647245    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"]
Nov 03 00:41:21.677813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.677776    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"]
Nov 03 00:41:21.677979 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.677886    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.687856 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.687831    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 00:41:21.688390 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.688364    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 00:41:21.688510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.688397    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 00:41:21.688510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.688416    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bs5h4g1ol8qg4\""
Nov 03 00:41:21.688510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.688461    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2rzvj\""
Nov 03 00:41:21.688510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.688494    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 00:41:21.759938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.759900    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf0dce65-0138-4e7d-8303-745ec180676a-audit-log\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.759955    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.760046    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-client-certs\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.760077    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-metrics-server-audit-profiles\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.760093    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krtdw\" (UniqueName: \"kubernetes.io/projected/cf0dce65-0138-4e7d-8303-745ec180676a-kube-api-access-krtdw\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760350 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.760131    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-client-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.760350 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.760290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-tls\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860713 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860713 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860715    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-client-certs\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860967 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-metrics-server-audit-profiles\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860967 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860771    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krtdw\" (UniqueName: \"kubernetes.io/projected/cf0dce65-0138-4e7d-8303-745ec180676a-kube-api-access-krtdw\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860967 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860805    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-client-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860967 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860839    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-tls\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.860967 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.860904    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf0dce65-0138-4e7d-8303-745ec180676a-audit-log\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.861629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.861602    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.861941 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.861918    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf0dce65-0138-4e7d-8303-745ec180676a-metrics-server-audit-profiles\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.862038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.862014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf0dce65-0138-4e7d-8303-745ec180676a-audit-log\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.863810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.863788    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-tls\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.863959 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.863940    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-secret-metrics-server-client-certs\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.864404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.864386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf0dce65-0138-4e7d-8303-745ec180676a-client-ca-bundle\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:21.874141 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:21.874117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krtdw\" (UniqueName: \"kubernetes.io/projected/cf0dce65-0138-4e7d-8303-745ec180676a-kube-api-access-krtdw\") pod \"metrics-server-55f7ccd4c6-4tq9l\" (UID: \"cf0dce65-0138-4e7d-8303-745ec180676a\") " pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:22.007778 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.007698    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:22.324814 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.324780    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"]
Nov 03 00:41:22.328257 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:22.328229    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0dce65_0138_4e7d_8303_745ec180676a.slice/crio-69e76468e3f1fc515a2c95936c55922d6939e5862532b7782c8dca4ad0676b2b WatchSource:0}: Error finding container 69e76468e3f1fc515a2c95936c55922d6939e5862532b7782c8dca4ad0676b2b: Status 404 returned error can't find the container with id 69e76468e3f1fc515a2c95936c55922d6939e5862532b7782c8dca4ad0676b2b
Nov 03 00:41:22.490015 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.489981    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b94879f6d-6gw74"]
Nov 03 00:41:22.517410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.517373    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b94879f6d-6gw74"]
Nov 03 00:41:22.517602 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.517511    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.525261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.525228    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 00:41:22.525393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.525382    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mtbsr\""
Nov 03 00:41:22.527097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.527075    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 00:41:22.527231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.527150    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 00:41:22.527231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.527223    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 00:41:22.527347 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.527259    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 00:41:22.528562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.528543    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 00:41:22.567493 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgn5\" (UniqueName: \"kubernetes.io/projected/23a3d848-e0a8-4002-b281-006764a14e42-kube-api-access-xvgn5\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567665 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567521    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567665 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-serving-certs-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567761 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567664    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-metrics-client-ca\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567803 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567750    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567803 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567885 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567830    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.567918 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.567890    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-federate-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668292    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgn5\" (UniqueName: \"kubernetes.io/projected/23a3d848-e0a8-4002-b281-006764a14e42-kube-api-access-xvgn5\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668351    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-serving-certs-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668403    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-metrics-client-ca\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668455    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668506    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.668655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.668531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-federate-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.669868 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.669840    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-metrics-client-ca\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.670386 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.670367    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-serving-certs-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.671190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.671147    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.671808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.671759    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-federate-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.671808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.671789    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.671972 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.671819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-secret-telemeter-client\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.672554 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.672513    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/23a3d848-e0a8-4002-b281-006764a14e42-telemeter-client-tls\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.697943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.697915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgn5\" (UniqueName: \"kubernetes.io/projected/23a3d848-e0a8-4002-b281-006764a14e42-kube-api-access-xvgn5\") pod \"telemeter-client-6b94879f6d-6gw74\" (UID: \"23a3d848-e0a8-4002-b281-006764a14e42\") " pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:22.775514 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.775480    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cbkz7"
Nov 03 00:41:22.869531 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:22.869499    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74"
Nov 03 00:41:23.052234 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:23.052200    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l" event={"ID":"cf0dce65-0138-4e7d-8303-745ec180676a","Type":"ContainerStarted","Data":"69e76468e3f1fc515a2c95936c55922d6939e5862532b7782c8dca4ad0676b2b"}
Nov 03 00:41:23.176971 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:23.176942    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b94879f6d-6gw74"]
Nov 03 00:41:23.240027 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:23.239958    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a3d848_e0a8_4002_b281_006764a14e42.slice/crio-1866fb24c4a430cb76706eddeae116b3d2aefcbdf4d50b46ff610fb11ee8fa98 WatchSource:0}: Error finding container 1866fb24c4a430cb76706eddeae116b3d2aefcbdf4d50b46ff610fb11ee8fa98: Status 404 returned error can't find the container with id 1866fb24c4a430cb76706eddeae116b3d2aefcbdf4d50b46ff610fb11ee8fa98
Nov 03 00:41:24.060125 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.060043    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"}
Nov 03 00:41:24.060543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.060127    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"}
Nov 03 00:41:24.060543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.060238    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"}
Nov 03 00:41:24.060543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.060282    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"}
Nov 03 00:41:24.060543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.060320    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"}
Nov 03 00:41:24.061574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:24.061548    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74" event={"ID":"23a3d848-e0a8-4002-b281-006764a14e42","Type":"ContainerStarted","Data":"1866fb24c4a430cb76706eddeae116b3d2aefcbdf4d50b46ff610fb11ee8fa98"}
Nov 03 00:41:25.067088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:25.067015    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l" event={"ID":"cf0dce65-0138-4e7d-8303-745ec180676a","Type":"ContainerStarted","Data":"b7578fa8b855c4f2acf9a6a732958ed798af924b292cb6a57788cfb2d66ec7c1"}
Nov 03 00:41:26.075787 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:26.075760    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerStarted","Data":"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"}
Nov 03 00:41:26.112664 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:26.112619    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l" podStartSLOduration=2.796853537 podStartE2EDuration="5.112603837s" totalImagesPullingTime="2.3157503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:25.096551676 +0000 UTC m=+30.310295347" watchObservedRunningTime="2026-11-03 00:41:26.112603837 +0000 UTC m=+31.326347508"
Nov 03 00:41:26.112990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:26.112951    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.994461636 podStartE2EDuration="8.112942783s" totalImagesPullingTime="5.105610488s" totalInitContainerRuntime="12.870659ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:26.111027366 +0000 UTC m=+31.324771036" watchObservedRunningTime="2026-11-03 00:41:26.112942783 +0000 UTC m=+31.326686459"
Nov 03 00:41:27.051594 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:27.051566    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9g7gh"
Nov 03 00:41:27.082405 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:27.082375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74" event={"ID":"23a3d848-e0a8-4002-b281-006764a14e42","Type":"ContainerStarted","Data":"a2d95a310663bc82abcc61e210fa6825911d4d1e72ab1384c439488c1c8a37cb"}
Nov 03 00:41:27.082778 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:27.082423    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74" event={"ID":"23a3d848-e0a8-4002-b281-006764a14e42","Type":"ContainerStarted","Data":"534d0d92dafc61c3da32002b75a7fb626f7e7763b612346a45666c9f674f13c6"}
Nov 03 00:41:27.082778 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:27.082442    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74" event={"ID":"23a3d848-e0a8-4002-b281-006764a14e42","Type":"ContainerStarted","Data":"631c52b753eb06b42f1920e3b4544cc49c5a0e0d61a77ac0a675dff477e4eaf8"}
Nov 03 00:41:27.115525 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:27.115476    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b94879f6d-6gw74" podStartSLOduration=2.33974147 podStartE2EDuration="5.115459425s" totalImagesPullingTime="2.775717955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:27.114104671 +0000 UTC m=+32.327848355" watchObservedRunningTime="2026-11-03 00:41:27.115459425 +0000 UTC m=+32.329203098"
Nov 03 00:41:28.478309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.478276    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:41:28.481120 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.481103    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.488280 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.488259    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 00:41:28.488376 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.488277    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 00:41:28.488376 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.488284    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-94ctt\""
Nov 03 00:41:28.502614 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.502584    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:41:28.521941 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.521912    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 00:41:28.522076 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.521998    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 00:41:28.522178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.522151    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 00:41:28.522312 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.522296    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 00:41:28.522682 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.522668    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 00:41:28.525973 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.525954    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 00:41:28.619821 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619787    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.619821 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619822    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619849    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619960    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4qh\" (UniqueName: \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.619984    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.620007    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.620240 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.620046    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720475 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720475 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720475    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720496    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4qh\" (UniqueName: \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720518    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720569    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720633    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.720879 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.720660    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.721040 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.721013    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.721405 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.721384    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.721482 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.721441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.721482 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.721452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.721634 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.721616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.723210 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.723189    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.723291 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.723209    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.729968 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.729924    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4qh\" (UniqueName: \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\") pod \"console-64c678fdd-w5b78\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") " pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:28.791952 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:28.791913    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:29.066200 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:29.066172    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:41:29.068754 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:41:29.068726    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode154ab53_ee26_4a81_93d2_a89f7e777596.slice/crio-05f83c4932e65b5c195b34c252f8fc9d353b6ad152f4696d558f1c6a41e11a54 WatchSource:0}: Error finding container 05f83c4932e65b5c195b34c252f8fc9d353b6ad152f4696d558f1c6a41e11a54: Status 404 returned error can't find the container with id 05f83c4932e65b5c195b34c252f8fc9d353b6ad152f4696d558f1c6a41e11a54
Nov 03 00:41:29.089404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:29.089377    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c678fdd-w5b78" event={"ID":"e154ab53-ee26-4a81-93d2-a89f7e777596","Type":"ContainerStarted","Data":"05f83c4932e65b5c195b34c252f8fc9d353b6ad152f4696d558f1c6a41e11a54"}
Nov 03 00:41:32.110844 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:32.110356    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c678fdd-w5b78" event={"ID":"e154ab53-ee26-4a81-93d2-a89f7e777596","Type":"ContainerStarted","Data":"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"}
Nov 03 00:41:32.165580 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:32.165527    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64c678fdd-w5b78" podStartSLOduration=1.3971865669999999 podStartE2EDuration="4.165513485s" totalImagesPullingTime="2.768326918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:32.16487611 +0000 UTC m=+37.378619794" watchObservedRunningTime="2026-11-03 00:41:32.165513485 +0000 UTC m=+37.379257158"
Nov 03 00:41:38.792485 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:38.792399    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:38.792485 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:38.792458    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:38.797059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:38.797039    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:39.138294 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:39.138269    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:41:39.876312 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:41:39.876276    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:41:42.007896 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.007854    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:42.008307 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.007931    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:41:42.333686 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.333652    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:41:42.334078 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.334034    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" podUID="27ebb569-3603-406a-9999-42f66e578212" containerName="insights-gathering" containerID="cri-o://fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9" gracePeriod=30
Nov 03 00:41:42.572305 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.572280    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:41:42.748768 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.748690    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") pod \"27ebb569-3603-406a-9999-42f66e578212\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") "
Nov 03 00:41:42.748914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.748769    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") pod \"27ebb569-3603-406a-9999-42f66e578212\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") "
Nov 03 00:41:42.748914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.748812    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") pod \"27ebb569-3603-406a-9999-42f66e578212\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") "
Nov 03 00:41:42.749043 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749016    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") pod \"27ebb569-3603-406a-9999-42f66e578212\" (UID: \"27ebb569-3603-406a-9999-42f66e578212\") "
Nov 03 00:41:42.749152 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749117    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle" pod "27ebb569-3603-406a-9999-42f66e578212" (UID: "27ebb569-3603-406a-9999-42f66e578212"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:42.749298 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749188    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle" pod "27ebb569-3603-406a-9999-42f66e578212" (UID: "27ebb569-3603-406a-9999-42f66e578212"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:42.749364 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749348    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-trusted-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:41:42.749402 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749371    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27ebb569-3603-406a-9999-42f66e578212-service-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:41:42.749402 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.749349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path" pod "27ebb569-3603-406a-9999-42f66e578212" (UID: "27ebb569-3603-406a-9999-42f66e578212"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:42.751045 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.751024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc" pod "27ebb569-3603-406a-9999-42f66e578212" (UID: "27ebb569-3603-406a-9999-42f66e578212"). InnerVolumeSpecName "kube-api-access-tbdcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:42.850647 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.850613    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/27ebb569-3603-406a-9999-42f66e578212-archives-path\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:41:42.850647 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:42.850641    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdcc\" (UniqueName: \"kubernetes.io/projected/27ebb569-3603-406a-9999-42f66e578212-kube-api-access-tbdcc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:41:43.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.148284    4531 generic.go:356] "Generic (PLEG): container finished" podID="27ebb569-3603-406a-9999-42f66e578212" containerID="fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9" exitCode=2
Nov 03 00:41:43.148690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.148327    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v"
Nov 03 00:41:43.148690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.148331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" event={"ID":"27ebb569-3603-406a-9999-42f66e578212","Type":"ContainerDied","Data":"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"}
Nov 03 00:41:43.148690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.148443    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wj6b6-dbm2v" event={"ID":"27ebb569-3603-406a-9999-42f66e578212","Type":"ContainerDied","Data":"e8cff32d2a9578579237b978b185952a045790e644e6b0dee0ca29289299a1e0"}
Nov 03 00:41:43.148690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.148466    4531 scope.go:122] "RemoveContainer" containerID="fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"
Nov 03 00:41:43.156486 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.156468    4531 scope.go:122] "RemoveContainer" containerID="fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"
Nov 03 00:41:43.156749 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:41:43.156730    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9\": container with ID starting with fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9 not found: ID does not exist" containerID="fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"
Nov 03 00:41:43.156802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.156758    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9"} err="failed to get container status \"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9\": rpc error: code = NotFound desc = could not find container \"fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9\": container with ID starting with fa301ec2e87d3a64adaf3bc782f3abde4bd5e9b5a4c16863e2f47accce38b1b9 not found: ID does not exist"
Nov 03 00:41:43.166174 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.166138    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:41:43.172838 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:43.172819    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-wj6b6-dbm2v"]
Nov 03 00:41:44.899568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:41:44.899535    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ebb569-3603-406a-9999-42f66e578212" path="/var/lib/kubelet/pods/27ebb569-3603-406a-9999-42f66e578212/volumes"
Nov 03 00:42:02.013363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:02.013335    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:42:02.017212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:02.017157    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55f7ccd4c6-4tq9l"
Nov 03 00:42:04.625246 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.625213    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c755ffdd7-8wlk5"]
Nov 03 00:42:04.625651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.625489    4531 cpu_manager.go:414] "containerMap: removing container" podUID="27ebb569-3603-406a-9999-42f66e578212" containerName="insights-gathering"
Nov 03 00:42:04.625651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.625500    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebb569-3603-406a-9999-42f66e578212" containerName="insights-gathering"
Nov 03 00:42:04.625651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.625551    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebb569-3603-406a-9999-42f66e578212" containerName="insights-gathering"
Nov 03 00:42:04.634033 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.634008    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.646802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.646775    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c755ffdd7-8wlk5"]
Nov 03 00:42:04.703407 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e755136-002b-4519-8ebe-d56ce76b965b-tmp\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-service-ca\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703439    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/0e755136-002b-4519-8ebe-d56ce76b965b-kube-api-access-q28w8\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703508    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-oauth-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703721 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703628    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-trusted-ca-bundle\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703721 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703668    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703804 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-oauth-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.703804 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.703795    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-console-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804328    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804368    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-oauth-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-console-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e755136-002b-4519-8ebe-d56ce76b965b-tmp\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804571    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-service-ca\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804602    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/0e755136-002b-4519-8ebe-d56ce76b965b-kube-api-access-q28w8\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-oauth-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804688    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-trusted-ca-bundle\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.804904 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.804852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e755136-002b-4519-8ebe-d56ce76b965b-tmp\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.805156 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.805133    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-console-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.805273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.805156    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-oauth-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.805273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.805259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-service-ca\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.805513 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.805491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e755136-002b-4519-8ebe-d56ce76b965b-trusted-ca-bundle\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.806944 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.806927    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-serving-cert\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.807030 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.807013    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e755136-002b-4519-8ebe-d56ce76b965b-console-oauth-config\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.813795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.813770    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28w8\" (UniqueName: \"kubernetes.io/projected/0e755136-002b-4519-8ebe-d56ce76b965b-kube-api-access-q28w8\") pod \"console-6c755ffdd7-8wlk5\" (UID: \"0e755136-002b-4519-8ebe-d56ce76b965b\") " pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:04.945569 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:04.945491    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:05.229231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:05.229202    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c755ffdd7-8wlk5"]
Nov 03 00:42:05.231459 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:42:05.231433    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e755136_002b_4519_8ebe_d56ce76b965b.slice/crio-809b8039dd3cc7872decbed4b4dccbb3d541b60106ab86a801e660fa9ece25b5 WatchSource:0}: Error finding container 809b8039dd3cc7872decbed4b4dccbb3d541b60106ab86a801e660fa9ece25b5: Status 404 returned error can't find the container with id 809b8039dd3cc7872decbed4b4dccbb3d541b60106ab86a801e660fa9ece25b5
Nov 03 00:42:06.228373 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:06.228336    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c755ffdd7-8wlk5" event={"ID":"0e755136-002b-4519-8ebe-d56ce76b965b","Type":"ContainerStarted","Data":"d68dc852e55ac83a93df1f31358bc153b22671e00bba9c17e9f489d636750396"}
Nov 03 00:42:06.228764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:06.228388    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c755ffdd7-8wlk5" event={"ID":"0e755136-002b-4519-8ebe-d56ce76b965b","Type":"ContainerStarted","Data":"809b8039dd3cc7872decbed4b4dccbb3d541b60106ab86a801e660fa9ece25b5"}
Nov 03 00:42:06.250795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:06.250741    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c755ffdd7-8wlk5" podStartSLOduration=2.250728953 podStartE2EDuration="2.250728953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:06.249227779 +0000 UTC m=+71.462971448" watchObservedRunningTime="2026-11-03 00:42:06.250728953 +0000 UTC m=+71.464472626"
Nov 03 00:42:07.735277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735243    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:07.735674 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735505    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="alertmanager" containerID="cri-o://062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68" gracePeriod=120
Nov 03 00:42:07.735674 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735596    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-web" containerID="cri-o://0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9" gracePeriod=120
Nov 03 00:42:07.735674 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735636    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="prom-label-proxy" containerID="cri-o://b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1" gracePeriod=120
Nov 03 00:42:07.735835 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735632    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="config-reloader" containerID="cri-o://9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df" gracePeriod=120
Nov 03 00:42:07.735835 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735578    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-metric" containerID="cri-o://0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729" gracePeriod=120
Nov 03 00:42:07.735835 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:07.735640    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy" containerID="cri-o://3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7" gracePeriod=120
Nov 03 00:42:08.239325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239294    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1" exitCode=0
Nov 03 00:42:08.239325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239321    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7" exitCode=0
Nov 03 00:42:08.239325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239327    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df" exitCode=0
Nov 03 00:42:08.239325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239333    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68" exitCode=0
Nov 03 00:42:08.239575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239356    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"}
Nov 03 00:42:08.239575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239406    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"}
Nov 03 00:42:08.239575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239431    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"}
Nov 03 00:42:08.239575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.239454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"}
Nov 03 00:42:08.969811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:08.969782    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.035142 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035099    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035142 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035138    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035177    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035236    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035271    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035297    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035341    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035385    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035432    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035482    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035510    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.035860 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035804    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.035916 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.035853    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:09.036355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.036327    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:09.038823 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.038770    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.039094 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039067    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.039094 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "kube-api-access-dlks6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:09.039313 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039273    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.039436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.039914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039882    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.040004 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.039969    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:09.043614 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.043594    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.049471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.049445    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.136204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136104    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\") pod \"70f572ed-3c18-4090-a0e1-289fbdc229bf\" (UID: \"70f572ed-3c18-4090-a0e1-289fbdc229bf\") "
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136297    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-metrics-client-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136311    4531 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-out\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136320    4531 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-tls-assets\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136329    4531 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-web-config\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136337    4531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-main-db\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136347    4531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70f572ed-3c18-4090-a0e1-289fbdc229bf-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136355    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlks6\" (UniqueName: \"kubernetes.io/projected/70f572ed-3c18-4090-a0e1-289fbdc229bf-kube-api-access-dlks6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136364    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136374    4531 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-cluster-tls-config\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136383    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136391    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-main-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.136737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.136402    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.138066 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.138044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume" pod "70f572ed-3c18-4090-a0e1-289fbdc229bf" (UID: "70f572ed-3c18-4090-a0e1-289fbdc229bf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.237126 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.237090    4531 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70f572ed-3c18-4090-a0e1-289fbdc229bf-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:09.246028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.245998    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729" exitCode=0
Nov 03 00:42:09.246028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246021    4531 generic.go:356] "Generic (PLEG): container finished" podID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerID="0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9" exitCode=0
Nov 03 00:42:09.246208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246051    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"}
Nov 03 00:42:09.246208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246076    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.246208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246107    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"}
Nov 03 00:42:09.246208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246133    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70f572ed-3c18-4090-a0e1-289fbdc229bf","Type":"ContainerDied","Data":"71cb5cc2eecc67826c9639034aa3173f8f68bd8e41149c24d415446799d77b08"}
Nov 03 00:42:09.246208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.246177    4531 scope.go:122] "RemoveContainer" containerID="b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"
Nov 03 00:42:09.253714 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.253698    4531 scope.go:122] "RemoveContainer" containerID="0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"
Nov 03 00:42:09.260739 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.260723    4531 scope.go:122] "RemoveContainer" containerID="3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"
Nov 03 00:42:09.267738 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.267720    4531 scope.go:122] "RemoveContainer" containerID="0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"
Nov 03 00:42:09.271113 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.271091    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:09.274825 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.274809    4531 scope.go:122] "RemoveContainer" containerID="9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"
Nov 03 00:42:09.281363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.281348    4531 scope.go:122] "RemoveContainer" containerID="062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"
Nov 03 00:42:09.284270 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.284247    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:09.288791 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.288767    4531 scope.go:122] "RemoveContainer" containerID="0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"
Nov 03 00:42:09.295435 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.295419    4531 scope.go:122] "RemoveContainer" containerID="b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"
Nov 03 00:42:09.295719 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.295700    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1\": container with ID starting with b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1 not found: ID does not exist" containerID="b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"
Nov 03 00:42:09.295764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.295730    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"} err="failed to get container status \"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1\": rpc error: code = NotFound desc = could not find container \"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1\": container with ID starting with b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1 not found: ID does not exist"
Nov 03 00:42:09.295764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.295748    4531 scope.go:122] "RemoveContainer" containerID="0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"
Nov 03 00:42:09.295973 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.295954    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729\": container with ID starting with 0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729 not found: ID does not exist" containerID="0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"
Nov 03 00:42:09.296013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.295980    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"} err="failed to get container status \"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729\": rpc error: code = NotFound desc = could not find container \"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729\": container with ID starting with 0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729 not found: ID does not exist"
Nov 03 00:42:09.296013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.295997    4531 scope.go:122] "RemoveContainer" containerID="3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"
Nov 03 00:42:09.296264 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.296239    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7\": container with ID starting with 3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7 not found: ID does not exist" containerID="3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"
Nov 03 00:42:09.296313 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296261    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"} err="failed to get container status \"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7\": rpc error: code = NotFound desc = could not find container \"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7\": container with ID starting with 3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7 not found: ID does not exist"
Nov 03 00:42:09.296313 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296274    4531 scope.go:122] "RemoveContainer" containerID="0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"
Nov 03 00:42:09.296484 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.296470    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9\": container with ID starting with 0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9 not found: ID does not exist" containerID="0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"
Nov 03 00:42:09.296521 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296486    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"} err="failed to get container status \"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9\": rpc error: code = NotFound desc = could not find container \"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9\": container with ID starting with 0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9 not found: ID does not exist"
Nov 03 00:42:09.296521 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296498    4531 scope.go:122] "RemoveContainer" containerID="9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"
Nov 03 00:42:09.296668 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.296654    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df\": container with ID starting with 9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df not found: ID does not exist" containerID="9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"
Nov 03 00:42:09.296703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296670    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"} err="failed to get container status \"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df\": rpc error: code = NotFound desc = could not find container \"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df\": container with ID starting with 9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df not found: ID does not exist"
Nov 03 00:42:09.296703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296680    4531 scope.go:122] "RemoveContainer" containerID="062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"
Nov 03 00:42:09.296876 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.296864    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68\": container with ID starting with 062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68 not found: ID does not exist" containerID="062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"
Nov 03 00:42:09.296911 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296878    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"} err="failed to get container status \"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68\": rpc error: code = NotFound desc = could not find container \"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68\": container with ID starting with 062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68 not found: ID does not exist"
Nov 03 00:42:09.296911 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.296890    4531 scope.go:122] "RemoveContainer" containerID="0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"
Nov 03 00:42:09.297081 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.297062    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68\": container with ID starting with 0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68 not found: ID does not exist" containerID="0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"
Nov 03 00:42:09.297119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297087    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"} err="failed to get container status \"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68\": rpc error: code = NotFound desc = could not find container \"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68\": container with ID starting with 0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68 not found: ID does not exist"
Nov 03 00:42:09.297119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297107    4531 scope.go:122] "RemoveContainer" containerID="b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"
Nov 03 00:42:09.297336 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297319    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1"} err="failed to get container status \"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1\": rpc error: code = NotFound desc = could not find container \"b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1\": container with ID starting with b02c95ee182dd4bdcbb3157b02b09b35a0f6a2b3da5468f0fc733160d1ad0bc1 not found: ID does not exist"
Nov 03 00:42:09.297336 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297335    4531 scope.go:122] "RemoveContainer" containerID="0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"
Nov 03 00:42:09.297493 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297480    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729"} err="failed to get container status \"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729\": rpc error: code = NotFound desc = could not find container \"0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729\": container with ID starting with 0fa6683ce7e930eb0d9f46bbc32b43a3b1da8fb53a823c80253f32103810e729 not found: ID does not exist"
Nov 03 00:42:09.297526 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297492    4531 scope.go:122] "RemoveContainer" containerID="3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"
Nov 03 00:42:09.297670 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297654    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7"} err="failed to get container status \"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7\": rpc error: code = NotFound desc = could not find container \"3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7\": container with ID starting with 3dd9262c4ea86b62127045b94e98d8397e8789ca40158a81069ae9d0f832b7b7 not found: ID does not exist"
Nov 03 00:42:09.297707 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297669    4531 scope.go:122] "RemoveContainer" containerID="0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"
Nov 03 00:42:09.297913 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297890    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9"} err="failed to get container status \"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9\": rpc error: code = NotFound desc = could not find container \"0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9\": container with ID starting with 0cabf49c573fe17eccc9a25b56495fc924ab684c2b1fa59c861067a785c264f9 not found: ID does not exist"
Nov 03 00:42:09.297913 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.297911    4531 scope.go:122] "RemoveContainer" containerID="9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"
Nov 03 00:42:09.298130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.298115    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df"} err="failed to get container status \"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df\": rpc error: code = NotFound desc = could not find container \"9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df\": container with ID starting with 9954e17b7133713a3441c56a3d766b5a462d1f3ead6145572e3a23e586cfc2df not found: ID does not exist"
Nov 03 00:42:09.298194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.298131    4531 scope.go:122] "RemoveContainer" containerID="062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"
Nov 03 00:42:09.298358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.298343    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68"} err="failed to get container status \"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68\": rpc error: code = NotFound desc = could not find container \"062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68\": container with ID starting with 062389a1d41ecd36a36b632793e53fa32de0063b73f646d66911dc4f13cdfb68 not found: ID does not exist"
Nov 03 00:42:09.298358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.298358    4531 scope.go:122] "RemoveContainer" containerID="0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"
Nov 03 00:42:09.298597 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.298580    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68"} err="failed to get container status \"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68\": rpc error: code = NotFound desc = could not find container \"0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68\": container with ID starting with 0b57de7dda2d0563e1e0fd9b6780dc5cef348207177d18497c06bce967fdfc68 not found: ID does not exist"
Nov 03 00:42:09.322986 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.322957    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:09.323271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323258    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-metric"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323273    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-metric"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323289    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="alertmanager"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323294    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="alertmanager"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323301    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323306    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323315    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="init-config-reloader"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323320    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="init-config-reloader"
Nov 03 00:42:09.323326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323326    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-web"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323331    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-web"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323338    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="prom-label-proxy"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323343    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="prom-label-proxy"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323350    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="config-reloader"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323354    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="config-reloader"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323401    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-metric"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323410    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="alertmanager"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323414    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323420    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="kube-rbac-proxy-web"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323425    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="prom-label-proxy"
Nov 03 00:42:09.323572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.323430    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" containerName="config-reloader"
Nov 03 00:42:09.328125 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.328108    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.333112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.333089    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 00:42:09.333112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.333106    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 00:42:09.334368 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.334345    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 00:42:09.334476 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.334396    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 00:42:09.334813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.334795    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 00:42:09.334887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.334828    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 00:42:09.335118 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.335104    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 00:42:09.335118 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.335109    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qswlh\""
Nov 03 00:42:09.335234 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.335111    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 00:42:09.337442 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337425    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337548 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337463    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337548 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337491    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337628 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337575    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337628 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337612    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvv4\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-kube-api-access-7dvv4\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337687 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337659    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337687 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337677    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337760 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337694    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337760 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337712    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-out\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337865 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337922 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337877    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.337922 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337911    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-web-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.338025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.337965    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.338639 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.338624    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 00:42:09.350843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.350824    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:09.439339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439260    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439312    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-out\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439349    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439413    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-web-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439444    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439496    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439518    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439545    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439591    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvv4\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-kube-api-access-7dvv4\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439624    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.439943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.439928    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.440637 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.440323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.441127 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.441069    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442621 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-out\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442621 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442404    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442621 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442561    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-web-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442621 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442593    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442844 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442666    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.442955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.442933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.443299 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.443282    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.443838 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.443818    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.443968 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.443951    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.450223 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.450205    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvv4\" (UniqueName: \"kubernetes.io/projected/9c5afd5b-b33b-4a82-bc4c-b31e4c098be0-kube-api-access-7dvv4\") pod \"alertmanager-main-0\" (UID: \"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.652556 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.652518    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:42:09.875049 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:09.875018    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:42:09.934714 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:09.934674    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:42:09.940542 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:42:09.940511    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5afd5b_b33b_4a82_bc4c_b31e4c098be0.slice/crio-fec1fe9d1402c6436b5a30a4f102ae0bf24ba0e50118b164fa0ab11aef8252bc WatchSource:0}: Error finding container fec1fe9d1402c6436b5a30a4f102ae0bf24ba0e50118b164fa0ab11aef8252bc: Status 404 returned error can't find the container with id fec1fe9d1402c6436b5a30a4f102ae0bf24ba0e50118b164fa0ab11aef8252bc
Nov 03 00:42:10.251383 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:10.251299    4531 generic.go:356] "Generic (PLEG): container finished" podID="9c5afd5b-b33b-4a82-bc4c-b31e4c098be0" containerID="45ac2bfe594a0b6fdc634d30efa55b408112501735ab41f4f4206b55e731c15f" exitCode=0
Nov 03 00:42:10.251383 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:10.251339    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerDied","Data":"45ac2bfe594a0b6fdc634d30efa55b408112501735ab41f4f4206b55e731c15f"}
Nov 03 00:42:10.251767 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:10.251389    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"fec1fe9d1402c6436b5a30a4f102ae0bf24ba0e50118b164fa0ab11aef8252bc"}
Nov 03 00:42:10.900264 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:10.900234    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f572ed-3c18-4090-a0e1-289fbdc229bf" path="/var/lib/kubelet/pods/70f572ed-3c18-4090-a0e1-289fbdc229bf/volumes"
Nov 03 00:42:11.258587 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258509    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"e8720942939d0be4dc5941dde4f02dd7fc156bbf30fe0fdaf56a4bc572f001c5"}
Nov 03 00:42:11.258587 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258569    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"43a22811a3010d3b0e43cbe7a3c0522dda630bb9ee9fc5afba26b18d16a9b9ad"}
Nov 03 00:42:11.258969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258600    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"d1b953f88b8899b9d8bdf5df4516cac69c8f56146e7be3ebd34cdb6ae7d25901"}
Nov 03 00:42:11.258969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258625    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"4670ce372bbc2a03437f9cfed31d927a3318fef549967a60d4648b42b4eb4dce"}
Nov 03 00:42:11.258969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258648    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"580911528d1b823ace68a868a5b03b65e2d8782b95844839e00b1a7a5831f0cb"}
Nov 03 00:42:11.258969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.258672    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c5afd5b-b33b-4a82-bc4c-b31e4c098be0","Type":"ContainerStarted","Data":"c4ffc8aefced3806557d89ca2608d3508f35cb1b4b83ac10a31265f5dffca46f"}
Nov 03 00:42:11.291654 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:11.291600    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.278217034 podStartE2EDuration="2.291574139s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.357105ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:11.290316129 +0000 UTC m=+76.504059800" watchObservedRunningTime="2026-11-03 00:42:11.291574139 +0000 UTC m=+76.505317872"
Nov 03 00:42:14.946016 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:14.945982    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:14.946419 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:14.946038    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:14.951543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:14.951520    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:15.262546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.262462    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:42:15.262714 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.262652    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" podUID="04fea578-3e8d-45b3-966a-af5c5a08d0da" containerName="insights-gathering" containerID="cri-o://22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c" gracePeriod=30
Nov 03 00:42:15.277463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.277444    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c755ffdd7-8wlk5"
Nov 03 00:42:15.338774 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.338720    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:42:15.499765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.499739    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:42:15.587645 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.587620    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\" (UniqueName: \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\") pod \"04fea578-3e8d-45b3-966a-af5c5a08d0da\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") "
Nov 03 00:42:15.587799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.587654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\") pod \"04fea578-3e8d-45b3-966a-af5c5a08d0da\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") "
Nov 03 00:42:15.587799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.587678    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\") pod \"04fea578-3e8d-45b3-966a-af5c5a08d0da\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") "
Nov 03 00:42:15.587799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.587746    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\") pod \"04fea578-3e8d-45b3-966a-af5c5a08d0da\" (UID: \"04fea578-3e8d-45b3-966a-af5c5a08d0da\") "
Nov 03 00:42:15.588034 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.588009    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path" pod "04fea578-3e8d-45b3-966a-af5c5a08d0da" (UID: "04fea578-3e8d-45b3-966a-af5c5a08d0da"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:15.588091 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.588052    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle" pod "04fea578-3e8d-45b3-966a-af5c5a08d0da" (UID: "04fea578-3e8d-45b3-966a-af5c5a08d0da"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:15.588123 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.588085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle" pod "04fea578-3e8d-45b3-966a-af5c5a08d0da" (UID: "04fea578-3e8d-45b3-966a-af5c5a08d0da"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:15.589694 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.589670    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj" pod "04fea578-3e8d-45b3-966a-af5c5a08d0da" (UID: "04fea578-3e8d-45b3-966a-af5c5a08d0da"). InnerVolumeSpecName "kube-api-access-flghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:15.688810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.688771    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-service-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:15.688810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.688801    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flghj\" (UniqueName: \"kubernetes.io/projected/04fea578-3e8d-45b3-966a-af5c5a08d0da-kube-api-access-flghj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:15.688810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.688813    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/04fea578-3e8d-45b3-966a-af5c5a08d0da-archives-path\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:15.689024 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:15.688821    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04fea578-3e8d-45b3-966a-af5c5a08d0da-trusted-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:16.278252 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.278220    4531 generic.go:356] "Generic (PLEG): container finished" podID="04fea578-3e8d-45b3-966a-af5c5a08d0da" containerID="22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c" exitCode=2
Nov 03 00:42:16.278688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.278260    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dftf6-zs2wg"
Nov 03 00:42:16.278688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.278303    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" event={"ID":"04fea578-3e8d-45b3-966a-af5c5a08d0da","Type":"ContainerDied","Data":"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"}
Nov 03 00:42:16.278688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.278339    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dftf6-zs2wg" event={"ID":"04fea578-3e8d-45b3-966a-af5c5a08d0da","Type":"ContainerDied","Data":"e96341f37270457b78e42381264c5a081875d4ebed730b9b59db9f75bae4785e"}
Nov 03 00:42:16.278688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.278357    4531 scope.go:122] "RemoveContainer" containerID="22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"
Nov 03 00:42:16.287226 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.287205    4531 scope.go:122] "RemoveContainer" containerID="22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"
Nov 03 00:42:16.287576 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:16.287516    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c\": container with ID starting with 22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c not found: ID does not exist" containerID="22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"
Nov 03 00:42:16.287576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.287552    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c"} err="failed to get container status \"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c\": rpc error: code = NotFound desc = could not find container \"22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c\": container with ID starting with 22896a9bd6bd54c81682a5c0c87281d60741024cb33ea00f2569c7585674b69c not found: ID does not exist"
Nov 03 00:42:16.301490 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.301463    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:42:16.306438 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.306416    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-dftf6-zs2wg"]
Nov 03 00:42:16.900408 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:16.900369    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fea578-3e8d-45b3-966a-af5c5a08d0da" path="/var/lib/kubelet/pods/04fea578-3e8d-45b3-966a-af5c5a08d0da/volumes"
Nov 03 00:42:39.880185 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:39.876590    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:42:40.358643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.358609    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64c678fdd-w5b78" podUID="e154ab53-ee26-4a81-93d2-a89f7e777596" containerName="console" containerID="cri-o://4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174" gracePeriod=15
Nov 03 00:42:40.593828 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.593803    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:42:40.683893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.683805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.683893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.683857    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.683893    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.683944    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.683975    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\" (UniqueName: \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684073    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\") pod \"e154ab53-ee26-4a81-93d2-a89f7e777596\" (UID: \"e154ab53-ee26-4a81-93d2-a89f7e777596\") "
Nov 03 00:42:40.684450 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684363    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:40.684450 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684363    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:40.684556 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684434    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:40.684556 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684515    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:40.684733 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.684707    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:40.686335 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.686305    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "kube-api-access-8q4qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:40.686428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.686333    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:40.686428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.686317    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert" pod "e154ab53-ee26-4a81-93d2-a89f7e777596" (UID: "e154ab53-ee26-4a81-93d2-a89f7e777596"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:40.784755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784716    4531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-console-config\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.784755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784744    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-trusted-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.784755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784754    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q4qh\" (UniqueName: \"kubernetes.io/projected/e154ab53-ee26-4a81-93d2-a89f7e777596-kube-api-access-8q4qh\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.784755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784764    4531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-oauth-serving-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.785002 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784773    4531 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e154ab53-ee26-4a81-93d2-a89f7e777596-tmp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.785002 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784783    4531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-serving-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.785002 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784791    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e154ab53-ee26-4a81-93d2-a89f7e777596-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:40.785002 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:40.784801    4531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e154ab53-ee26-4a81-93d2-a89f7e777596-console-oauth-config\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:41.364702 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.364666    4531 generic.go:356] "Generic (PLEG): container finished" podID="e154ab53-ee26-4a81-93d2-a89f7e777596" containerID="4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174" exitCode=0
Nov 03 00:42:41.365115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.364718    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c678fdd-w5b78"
Nov 03 00:42:41.365115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.364750    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c678fdd-w5b78" event={"ID":"e154ab53-ee26-4a81-93d2-a89f7e777596","Type":"ContainerDied","Data":"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"}
Nov 03 00:42:41.365115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.364806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c678fdd-w5b78" event={"ID":"e154ab53-ee26-4a81-93d2-a89f7e777596","Type":"ContainerDied","Data":"05f83c4932e65b5c195b34c252f8fc9d353b6ad152f4696d558f1c6a41e11a54"}
Nov 03 00:42:41.365115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.364827    4531 scope.go:122] "RemoveContainer" containerID="4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"
Nov 03 00:42:41.372543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.372528    4531 scope.go:122] "RemoveContainer" containerID="4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"
Nov 03 00:42:41.372816 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:42:41.372796    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174\": container with ID starting with 4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174 not found: ID does not exist" containerID="4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"
Nov 03 00:42:41.372914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.372820    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174"} err="failed to get container status \"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174\": rpc error: code = NotFound desc = could not find container \"4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174\": container with ID starting with 4497ea2a22e71b3758edb0a74a2076d6f052041ded009c07572ed29e0caed174 not found: ID does not exist"
Nov 03 00:42:41.382900 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.382880    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:42:41.387759 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:41.387739    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64c678fdd-w5b78"]
Nov 03 00:42:42.900205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:42:42.900152    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e154ab53-ee26-4a81-93d2-a89f7e777596" path="/var/lib/kubelet/pods/e154ab53-ee26-4a81-93d2-a89f7e777596/volumes"
Nov 03 00:43:09.875862 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:43:09.875782    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:43:39.875946 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:43:39.875911    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:44:08.939592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939551    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"]
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939843    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e154ab53-ee26-4a81-93d2-a89f7e777596" containerName="console"
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939854    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e154ab53-ee26-4a81-93d2-a89f7e777596" containerName="console"
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939882    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04fea578-3e8d-45b3-966a-af5c5a08d0da" containerName="insights-gathering"
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939890    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fea578-3e8d-45b3-966a-af5c5a08d0da" containerName="insights-gathering"
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e154ab53-ee26-4a81-93d2-a89f7e777596" containerName="console"
Nov 03 00:44:08.939989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.939962    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fea578-3e8d-45b3-966a-af5c5a08d0da" containerName="insights-gathering"
Nov 03 00:44:08.943078 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.943056    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:08.948849 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.948826    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 00:44:08.948849 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.948842    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 00:44:08.949068 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.948860    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hcq9l\""
Nov 03 00:44:08.957749 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:08.957718    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"]
Nov 03 00:44:09.003519 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.003489    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"]
Nov 03 00:44:09.006855 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.006835    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.020288 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.020256    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"]
Nov 03 00:44:09.038634 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.038602    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-zwgjd\""
Nov 03 00:44:09.047937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.047910    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4vx\" (UniqueName: \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.048086 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.048069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.048152 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.048121    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.149468 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.149679 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149484    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4vx\" (UniqueName: \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.149679 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2sv\" (UniqueName: \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.149679 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149642    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.149807 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149712    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.149807 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.149757    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.150085 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.150071    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.150122 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.150101    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.159186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.159148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4vx\" (UniqueName: \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.250557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.250466    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2sv\" (UniqueName: \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.250557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.250515    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.250557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.250555    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.250930 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.250909    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.250970 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.250945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.259440 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.259408    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:09.265248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.265224    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2sv\" (UniqueName: \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.322747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.322718    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:09.540815 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.540735    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"]
Nov 03 00:44:09.544102 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:09.544071    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ccdda8_f3e3_4a8d_a317_30e2aaa94c74.slice/crio-15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97 WatchSource:0}: Error finding container 15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97: Status 404 returned error can't find the container with id 15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97
Nov 03 00:44:09.601839 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.601808    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"]
Nov 03 00:44:09.604883 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:09.604851    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f72592_4afa_4c81_b5c8_871ee26826f9.slice/crio-1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348 WatchSource:0}: Error finding container 1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348: Status 404 returned error can't find the container with id 1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348
Nov 03 00:44:09.654510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.654477    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerStarted","Data":"15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97"}
Nov 03 00:44:09.655633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:09.655604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt" event={"ID":"61f72592-4afa-4c81-b5c8-871ee26826f9","Type":"ContainerStarted","Data":"1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348"}
Nov 03 00:44:09.877550 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:44:09.877509    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:44:12.216005 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.214610    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"]
Nov 03 00:44:12.218280 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.218258    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.223440 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.223398    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-n589w\""
Nov 03 00:44:12.229150 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.229119    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"]
Nov 03 00:44:12.377826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.377781    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.378012 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.377936    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.378012 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.377981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrdc\" (UniqueName: \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.479843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.479121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.479843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.479220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.479843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.479247    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrdc\" (UniqueName: \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.479843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.479608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.479843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.479769    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.489095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.489060    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrdc\" (UniqueName: \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.538949 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.538908    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:12.823543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:12.823508    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"]
Nov 03 00:44:12.827846 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:12.827800    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460c8add_bf5e_41d0_a25d_ed561713040f.slice/crio-ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266 WatchSource:0}: Error finding container ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266: Status 404 returned error can't find the container with id ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266
Nov 03 00:44:13.513034 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.512990    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 00:44:13.518704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.517612    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.524534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.524508    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 00:44:13.524722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.524632    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 00:44:13.525277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.525235    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 00:44:13.525699 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.525680    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjnsk\""
Nov 03 00:44:13.533209 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.531620    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 00:44:13.611193 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.610880    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"]
Nov 03 00:44:13.615234 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.614908    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.625845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.625467    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-s482n\""
Nov 03 00:44:13.629117 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.629090    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"]
Nov 03 00:44:13.683612 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.683515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl" event={"ID":"460c8add-bf5e-41d0-a25d-ed561713040f","Type":"ContainerStarted","Data":"ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266"}
Nov 03 00:44:13.688558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688340    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688390    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688471    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688500    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688627    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688708    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5hp\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688736    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhd6\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-kube-api-access-nkhd6\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688791    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688818    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688860    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688885    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.688929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688916    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.689504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688962    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.689504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.688994    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.689504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.689018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789508 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789424    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789508 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789469    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789508 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789556    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789618    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789678    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789703    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.789790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789779    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.790356 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5hp\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.790356 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhd6\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-kube-api-access-nkhd6\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.790356 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789870    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.790356 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.789897    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.790356 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.790340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.790607 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.790543    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.790876    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.791441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.791537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.791606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.791767    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.792133    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.792197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.792747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.792644    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.793703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.793657    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.793703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.793678    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.795380 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.795347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.797101 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.797052    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.799989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.799954    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5hp\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.800425 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.800392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-skx44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.800645 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.800607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.802560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.802509    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhd6\" (UniqueName: \"kubernetes.io/projected/1c6afe96-ef02-4368-b078-6ec4b0e0e1ff-kube-api-access-nkhd6\") pod \"openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n\" (UID: \"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:13.842151 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.841284    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:13.934981 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:13.933523    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:14.144332 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.144217    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 00:44:14.149260 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:14.149225    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c6128_d29a_4fd2_a4b0_c01e2d068c44.slice/crio-b6b9f056d114fb8f9029434b35a67323ee57dd8e2f201a06267ee00b2f62ffd9 WatchSource:0}: Error finding container b6b9f056d114fb8f9029434b35a67323ee57dd8e2f201a06267ee00b2f62ffd9: Status 404 returned error can't find the container with id b6b9f056d114fb8f9029434b35a67323ee57dd8e2f201a06267ee00b2f62ffd9
Nov 03 00:44:14.225338 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.225298    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"]
Nov 03 00:44:14.228512 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:14.228470    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6afe96_ef02_4368_b078_6ec4b0e0e1ff.slice/crio-6c50e903b0b4c0192c6c450867b6d2e3cdba3d68be4ba09ac3faaa6648aa0d44 WatchSource:0}: Error finding container 6c50e903b0b4c0192c6c450867b6d2e3cdba3d68be4ba09ac3faaa6648aa0d44: Status 404 returned error can't find the container with id 6c50e903b0b4c0192c6c450867b6d2e3cdba3d68be4ba09ac3faaa6648aa0d44
Nov 03 00:44:14.360090 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.360054    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"]
Nov 03 00:44:14.365629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.365050    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.372758 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.372728    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"]
Nov 03 00:44:14.373891 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.373788    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-q875h\""
Nov 03 00:44:14.456130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.456049    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"]
Nov 03 00:44:14.459934 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.459908    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.464054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.463999    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bs74q\""
Nov 03 00:44:14.472091 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.472063    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"]
Nov 03 00:44:14.497204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.497144    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.497370 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.497275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.497436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.497374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgtk\" (UniqueName: \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.561807 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.561771    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"]
Nov 03 00:44:14.565672 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.565647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.574677 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.574651    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"]
Nov 03 00:44:14.576783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.574812    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5nxzs\""
Nov 03 00:44:14.598166 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598132    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.598318 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598199    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.598318 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598225    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsrt\" (UniqueName: \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.598318 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598252    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.598318 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgtk\" (UniqueName: \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.598544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598347    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.598719 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.598688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.599381 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.599359    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.611415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.611383    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgtk\" (UniqueName: \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.661143 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.661111    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"]
Nov 03 00:44:14.665795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.665771    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.673247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.673205    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-t6hvl\""
Nov 03 00:44:14.675195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.674749    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"]
Nov 03 00:44:14.690906 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.690873    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerStarted","Data":"b6b9f056d114fb8f9029434b35a67323ee57dd8e2f201a06267ee00b2f62ffd9"}
Nov 03 00:44:14.691042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.691020    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:14.692231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.692153    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n" event={"ID":"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff","Type":"ContainerStarted","Data":"6c50e903b0b4c0192c6c450867b6d2e3cdba3d68be4ba09ac3faaa6648aa0d44"}
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699424    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699496    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsrt\" (UniqueName: \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699576    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsgs\" (UniqueName: \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699609    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.699637    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.700026    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.700545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.700266    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.710654 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.710558    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsrt\" (UniqueName: \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.782963 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.782760    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:14.801094 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801000    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldjnd\" (UniqueName: \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.801279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801091    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.801279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801147    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsgs\" (UniqueName: \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.801279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801207    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.801279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801234    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.801279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.802008 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.801979    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.802298 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.802270    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.816108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.816000    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsgs\" (UniqueName: \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.888771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.888487    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:14.902567 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.902537    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.902703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.902600    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldjnd\" (UniqueName: \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.902703 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.902671    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.902978 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.902955    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.902978 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.902970    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.912654 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.912626    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldjnd\" (UniqueName: \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:14.988953 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.988913    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"]
Nov 03 00:44:14.989479 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:14.989454    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:15.101894 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.101860    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"]
Nov 03 00:44:15.111887 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:15.111852    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df50a03_8c4e_4062_9423_b7834fc1499c.slice/crio-2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22 WatchSource:0}: Error finding container 2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22: Status 404 returned error can't find the container with id 2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22
Nov 03 00:44:15.188402 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.188239    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"]
Nov 03 00:44:15.191509 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:15.191475    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaef1caf_7208_4282_b16e_ac5c24306dd1.slice/crio-c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c WatchSource:0}: Error finding container c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c: Status 404 returned error can't find the container with id c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c
Nov 03 00:44:15.289863 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.289784    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"]
Nov 03 00:44:15.299614 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:15.299580    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fea957c_d8e8_4ba8_9564_490b26a120e0.slice/crio-3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168 WatchSource:0}: Error finding container 3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168: Status 404 returned error can't find the container with id 3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168
Nov 03 00:44:15.705031 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.704905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw" event={"ID":"7df50a03-8c4e-4062-9423-b7834fc1499c","Type":"ContainerStarted","Data":"2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22"}
Nov 03 00:44:15.714574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.713423    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj" event={"ID":"d90cdbb9-26f1-493c-b2fe-7727021e903f","Type":"ContainerStarted","Data":"aa6e1eaf6adb4c83b173c0560c9c56eec97f63e0fef80efe292e165435141768"}
Nov 03 00:44:15.716354 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.716193    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw" event={"ID":"daef1caf-7208-4282-b16e-ac5c24306dd1","Type":"ContainerStarted","Data":"c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c"}
Nov 03 00:44:15.726069 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.725992    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb" event={"ID":"2fea957c-d8e8-4ba8-9564-490b26a120e0","Type":"ContainerStarted","Data":"3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168"}
Nov 03 00:44:15.853271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.853231    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"]
Nov 03 00:44:15.859949 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.859921    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:15.866677 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:15.866590    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"]
Nov 03 00:44:16.017931 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.016760    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.017931 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.016964    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfs7d\" (UniqueName: \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.017931 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.017013    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.119103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.117859    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfs7d\" (UniqueName: \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.119103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.117930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.119103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.118004    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.119103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.118426    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.119103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.119047    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.144088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.144032    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfs7d\" (UniqueName: \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.196751 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.196710    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:16.523829 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.523792    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"]
Nov 03 00:44:16.546211 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:16.546152    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab9c75c_be7d_4dfd_88e5_d4cc1af42b33.slice/crio-452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225 WatchSource:0}: Error finding container 452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225: Status 404 returned error can't find the container with id 452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225
Nov 03 00:44:16.749296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.749214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerStarted","Data":"452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225"}
Nov 03 00:44:16.886562 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.886518    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"]
Nov 03 00:44:16.891361 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.891333    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:16.905936 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:16.905912    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"]
Nov 03 00:44:17.031003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.030959    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.031213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.031045    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg56d\" (UniqueName: \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.031213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.031086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.131997 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.131943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.132210 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.132033    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg56d\" (UniqueName: \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.132210 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.132074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.132597 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.132575    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.132896 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.132823    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.161095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.161016    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg56d\" (UniqueName: \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.226536 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.226501    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:17.590903 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.590863    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"]
Nov 03 00:44:17.613219 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:17.613180    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6054ebb6_b128_4a18_9a5e_c6d41f2caeaa.slice/crio-ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0 WatchSource:0}: Error finding container ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0: Status 404 returned error can't find the container with id ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0
Nov 03 00:44:17.774793 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:17.774747    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk" event={"ID":"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa","Type":"ContainerStarted","Data":"ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0"}
Nov 03 00:44:20.818240 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:20.817915    4531 generic.go:356] "Generic (PLEG): container finished" podID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerID="bf6b9520a81046169a99b3c165c514bb9c69b8428a7a7041011c5eb29fac7afb" exitCode=0
Nov 03 00:44:20.818240 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:20.818010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerDied","Data":"bf6b9520a81046169a99b3c165c514bb9c69b8428a7a7041011c5eb29fac7afb"}
Nov 03 00:44:21.850986 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:21.850946    4531 generic.go:356] "Generic (PLEG): container finished" podID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerID="e6cd59da36568da3ac2ed6fb8f9730b8a5c32d8fd71984179dee5aed540fc94e" exitCode=0
Nov 03 00:44:21.851494 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:21.851017    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt" event={"ID":"61f72592-4afa-4c81-b5c8-871ee26826f9","Type":"ContainerDied","Data":"e6cd59da36568da3ac2ed6fb8f9730b8a5c32d8fd71984179dee5aed540fc94e"}
Nov 03 00:44:22.874696 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:22.872551    4531 generic.go:356] "Generic (PLEG): container finished" podID="460c8add-bf5e-41d0-a25d-ed561713040f" containerID="47f15c302b51145cf0124df96a19b8a961b67d040f724d289bd6d06bcf8d1c74" exitCode=0
Nov 03 00:44:22.874696 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:22.872701    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl" event={"ID":"460c8add-bf5e-41d0-a25d-ed561713040f","Type":"ContainerDied","Data":"47f15c302b51145cf0124df96a19b8a961b67d040f724d289bd6d06bcf8d1c74"}
Nov 03 00:44:23.365672 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:23.364915    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:23.365672 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:23.365019    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:23.365672 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:23.365064    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:23.904218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:23.902137    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerStarted","Data":"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"}
Nov 03 00:44:23.931647 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:23.930051    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podStartSLOduration=1.71838249 podStartE2EDuration="10.930032143s" totalImagesPullingTime="9.211649653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:23.92920675 +0000 UTC m=+209.142950423" watchObservedRunningTime="2026-11-03 00:44:23.930032143 +0000 UTC m=+209.143775816"
Nov 03 00:44:24.135342 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.135301    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:24.135466 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.135393    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:24.135466 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.135439    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:24.842247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.842207    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:24.848654 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.848630    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:24.926666 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.926624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n" event={"ID":"1c6afe96-ef02-4368-b078-6ec4b0e0e1ff","Type":"ContainerStarted","Data":"edbe592a0642d824782d6e36495c2e090ede27e43ae528a77f55cb0bf149c89d"}
Nov 03 00:44:24.935076 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.935045    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:24.935982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.935955    4531 generic.go:356] "Generic (PLEG): container finished" podID="460c8add-bf5e-41d0-a25d-ed561713040f" containerID="d1e7b95a4b55d0c6d31050f1564fa4fe0561d4ec1eb1d68de3ca6ac91e0cd12a" exitCode=0
Nov 03 00:44:24.937823 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.937804    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl" event={"ID":"460c8add-bf5e-41d0-a25d-ed561713040f","Type":"ContainerDied","Data":"d1e7b95a4b55d0c6d31050f1564fa4fe0561d4ec1eb1d68de3ca6ac91e0cd12a"}
Nov 03 00:44:24.937930 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.937842    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:24.939332 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.939315    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:44:24.953028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:24.951102    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n" podStartSLOduration=2.047819552 podStartE2EDuration="11.951084979s" totalImagesPullingTime="9.903265427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:24.949644227 +0000 UTC m=+210.163387901" watchObservedRunningTime="2026-11-03 00:44:24.951084979 +0000 UTC m=+210.164828655"
Nov 03 00:44:25.942143 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:25.942111    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:25.946058 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:25.946031    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:25.947559 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:25.947507    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n"
Nov 03 00:44:26.961023 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.960220    4531 generic.go:356] "Generic (PLEG): container finished" podID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerID="80fb7fc22cc2056f3c520737dc3e799038c02cca5013c9b3d6fcfaf175609560" exitCode=0
Nov 03 00:44:26.961023 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.960319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw" event={"ID":"7df50a03-8c4e-4062-9423-b7834fc1499c","Type":"ContainerDied","Data":"80fb7fc22cc2056f3c520737dc3e799038c02cca5013c9b3d6fcfaf175609560"}
Nov 03 00:44:26.965056 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.964374    4531 generic.go:356] "Generic (PLEG): container finished" podID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerID="72160823dd00123144ad659a9559bc36fe1b97fd3f64bb7e5a3f0fe6aed22972" exitCode=0
Nov 03 00:44:26.965056 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.964447    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj" event={"ID":"d90cdbb9-26f1-493c-b2fe-7727021e903f","Type":"ContainerDied","Data":"72160823dd00123144ad659a9559bc36fe1b97fd3f64bb7e5a3f0fe6aed22972"}
Nov 03 00:44:26.970251 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.970227    4531 generic.go:356] "Generic (PLEG): container finished" podID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerID="39c4811d32812b8683414ad25dc28abf7581ca69a8511ad6298b3fd46c1d444e" exitCode=0
Nov 03 00:44:26.970333 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:26.970317    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw" event={"ID":"daef1caf-7208-4282-b16e-ac5c24306dd1","Type":"ContainerDied","Data":"39c4811d32812b8683414ad25dc28abf7581ca69a8511ad6298b3fd46c1d444e"}
Nov 03 00:44:27.975557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.975520    4531 generic.go:356] "Generic (PLEG): container finished" podID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerID="b94f677cb187bcf128d0b75cef927b1b9e7d3cf043902a98120670240721091e" exitCode=0
Nov 03 00:44:27.975947 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.975578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk" event={"ID":"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa","Type":"ContainerDied","Data":"b94f677cb187bcf128d0b75cef927b1b9e7d3cf043902a98120670240721091e"}
Nov 03 00:44:27.977790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.977752    4531 generic.go:356] "Generic (PLEG): container finished" podID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerID="95fe68ce20cf5c6beb424c9ddfdb66a842832c1f06cefb2e4362915d9f787ec7" exitCode=0
Nov 03 00:44:27.977887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.977794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt" event={"ID":"61f72592-4afa-4c81-b5c8-871ee26826f9","Type":"ContainerDied","Data":"95fe68ce20cf5c6beb424c9ddfdb66a842832c1f06cefb2e4362915d9f787ec7"}
Nov 03 00:44:27.980025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.980007    4531 generic.go:356] "Generic (PLEG): container finished" podID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerID="d9b08e62d099e74778e19bc08e3231c2d31e1b32a15002bd83ae632415dad2ae" exitCode=0
Nov 03 00:44:27.980107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.980084    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerDied","Data":"d9b08e62d099e74778e19bc08e3231c2d31e1b32a15002bd83ae632415dad2ae"}
Nov 03 00:44:27.981496 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.981477    4531 generic.go:356] "Generic (PLEG): container finished" podID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerID="842cc7c5ed7cd481c3066213646bc240cf3dcb879d74aab7e51b162010d53e9e" exitCode=0
Nov 03 00:44:27.981575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.981540    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb" event={"ID":"2fea957c-d8e8-4ba8-9564-490b26a120e0","Type":"ContainerDied","Data":"842cc7c5ed7cd481c3066213646bc240cf3dcb879d74aab7e51b162010d53e9e"}
Nov 03 00:44:27.982945 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.982929    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerID="2bfb9751455a314feec649d59fe5584b3d77ec692695d20a384cc7e0529505e8" exitCode=0
Nov 03 00:44:27.983004 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:27.982955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerDied","Data":"2bfb9751455a314feec649d59fe5584b3d77ec692695d20a384cc7e0529505e8"}
Nov 03 00:44:28.989767 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.989730    4531 generic.go:356] "Generic (PLEG): container finished" podID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerID="e5c961f8212b2221ccd37310000f9d4ec145f0a3f0f6cf5b6c82a559426a1a13" exitCode=0
Nov 03 00:44:28.990245 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.989822    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk" event={"ID":"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa","Type":"ContainerDied","Data":"e5c961f8212b2221ccd37310000f9d4ec145f0a3f0f6cf5b6c82a559426a1a13"}
Nov 03 00:44:28.992495 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.992218    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerID="4f20540b3e5981791a6c1b0af41f619520448ccf3d9d63e3a2cb80eff1f52fbc" exitCode=0
Nov 03 00:44:28.992710 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.992686    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerDied","Data":"4f20540b3e5981791a6c1b0af41f619520448ccf3d9d63e3a2cb80eff1f52fbc"}
Nov 03 00:44:28.996496 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.996454    4531 generic.go:356] "Generic (PLEG): container finished" podID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerID="45d10be1062d08216dcf55060c78df469a9729b7c83c46110e00ac9eedd38c58" exitCode=0
Nov 03 00:44:28.996675 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.996648    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj" event={"ID":"d90cdbb9-26f1-493c-b2fe-7727021e903f","Type":"ContainerDied","Data":"45d10be1062d08216dcf55060c78df469a9729b7c83c46110e00ac9eedd38c58"}
Nov 03 00:44:28.999324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.999304    4531 generic.go:356] "Generic (PLEG): container finished" podID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerID="6b0de7af45df4f8fffea082bf3803e27c05d1000470222e5578e42de91d766db" exitCode=0
Nov 03 00:44:28.999402 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:28.999330    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw" event={"ID":"daef1caf-7208-4282-b16e-ac5c24306dd1","Type":"ContainerDied","Data":"6b0de7af45df4f8fffea082bf3803e27c05d1000470222e5578e42de91d766db"}
Nov 03 00:44:30.015326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:30.015119    4531 generic.go:356] "Generic (PLEG): container finished" podID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerID="bd20208a4697eaa0c5f337e23c4a070e7dde79b152f9c206b5e7ac57d10c4651" exitCode=0
Nov 03 00:44:30.015326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:30.015192    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw" event={"ID":"7df50a03-8c4e-4062-9423-b7834fc1499c","Type":"ContainerDied","Data":"bd20208a4697eaa0c5f337e23c4a070e7dde79b152f9c206b5e7ac57d10c4651"}
Nov 03 00:44:30.019752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:30.019725    4531 generic.go:356] "Generic (PLEG): container finished" podID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerID="fc05085676c2a96a5df4ee8be0dbfb866543b9c17eee9198a37dca9ca21ab1d9" exitCode=0
Nov 03 00:44:30.019863 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:30.019795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb" event={"ID":"2fea957c-d8e8-4ba8-9564-490b26a120e0","Type":"ContainerDied","Data":"fc05085676c2a96a5df4ee8be0dbfb866543b9c17eee9198a37dca9ca21ab1d9"}
Nov 03 00:44:39.909526 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:44:39.909396    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:44:43.183830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:43.183737    4531 generic.go:356] "Generic (PLEG): container finished" podID="460c8add-bf5e-41d0-a25d-ed561713040f" containerID="83ddc27ff8178e904df0d490eb23ad1737d2c99b94b31c195261bae8c4ed1f8b" exitCode=0
Nov 03 00:44:43.183830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:43.183809    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl" event={"ID":"460c8add-bf5e-41d0-a25d-ed561713040f","Type":"ContainerDied","Data":"83ddc27ff8178e904df0d490eb23ad1737d2c99b94b31c195261bae8c4ed1f8b"}
Nov 03 00:44:44.455129 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.454743    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:44.539919 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.539871    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\") pod \"460c8add-bf5e-41d0-a25d-ed561713040f\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") "
Nov 03 00:44:44.540107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.539939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\" (UniqueName: \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\") pod \"460c8add-bf5e-41d0-a25d-ed561713040f\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") "
Nov 03 00:44:44.540107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.539986    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\") pod \"460c8add-bf5e-41d0-a25d-ed561713040f\" (UID: \"460c8add-bf5e-41d0-a25d-ed561713040f\") "
Nov 03 00:44:44.541043 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.541014    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle" pod "460c8add-bf5e-41d0-a25d-ed561713040f" (UID: "460c8add-bf5e-41d0-a25d-ed561713040f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:44.544588 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.544543    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc" pod "460c8add-bf5e-41d0-a25d-ed561713040f" (UID: "460c8add-bf5e-41d0-a25d-ed561713040f"). InnerVolumeSpecName "kube-api-access-mwrdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:44.547242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.547216    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util" pod "460c8add-bf5e-41d0-a25d-ed561713040f" (UID: "460c8add-bf5e-41d0-a25d-ed561713040f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:44.641608 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.641566    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:44.641608 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.641610    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrdc\" (UniqueName: \"kubernetes.io/projected/460c8add-bf5e-41d0-a25d-ed561713040f-kube-api-access-mwrdc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:44.641861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:44.641629    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/460c8add-bf5e-41d0-a25d-ed561713040f-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:45.209653 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:45.209568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl" event={"ID":"460c8add-bf5e-41d0-a25d-ed561713040f","Type":"ContainerDied","Data":"ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266"}
Nov 03 00:44:45.209653 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:45.209630    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec04b98b375da99d29148ddb241888ca3f39baeb3a56dafc82a305820926b266"
Nov 03 00:44:45.209897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:45.209716    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tbvkl"
Nov 03 00:44:46.221503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:46.221463    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerStarted","Data":"c9eb65ea9227aa53dff100ae3d8d823bfbf46f663a186b9f56711a750b2e9b58"}
Nov 03 00:44:46.248212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:46.248112    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" podStartSLOduration=3.846093991 podStartE2EDuration="38.248092599s" totalImagesPullingTime="34.401998608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:46.246108589 +0000 UTC m=+231.459852261" watchObservedRunningTime="2026-11-03 00:44:46.248092599 +0000 UTC m=+231.461836275"
Nov 03 00:44:47.229709 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.229670    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerStarted","Data":"23c064d54ecf4e7375bd5903c90ed27456cd588f93c61672b1acbef05d5b8b48"}
Nov 03 00:44:47.232090 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.232060    4531 generic.go:356] "Generic (PLEG): container finished" podID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerID="68e48ce2ceb9d0aa2385ef5877e2f6cf79e1bc0be5f56dc37c8d8d6d69bcff72" exitCode=0
Nov 03 00:44:47.232262 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.232140    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt" event={"ID":"61f72592-4afa-4c81-b5c8-871ee26826f9","Type":"ContainerDied","Data":"68e48ce2ceb9d0aa2385ef5877e2f6cf79e1bc0be5f56dc37c8d8d6d69bcff72"}
Nov 03 00:44:47.234906 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.234883    4531 generic.go:356] "Generic (PLEG): container finished" podID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerID="c9eb65ea9227aa53dff100ae3d8d823bfbf46f663a186b9f56711a750b2e9b58" exitCode=0
Nov 03 00:44:47.235025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.234920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerDied","Data":"c9eb65ea9227aa53dff100ae3d8d823bfbf46f663a186b9f56711a750b2e9b58"}
Nov 03 00:44:47.273050 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:47.270724    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" podStartSLOduration=3.21899017 podStartE2EDuration="32.270709071s" totalImagesPullingTime="29.051718901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:47.251098723 +0000 UTC m=+232.464842395" watchObservedRunningTime="2026-11-03 00:44:47.270709071 +0000 UTC m=+232.484452745"
Nov 03 00:44:48.240758 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.240684    4531 generic.go:356] "Generic (PLEG): container finished" podID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerID="4bfaaf2f9fcddbe6e7be27469db0c9a09366d6fe3f4c90a0bd20b23d5479af0f" exitCode=0
Nov 03 00:44:48.241143 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.240762    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk" event={"ID":"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa","Type":"ContainerDied","Data":"4bfaaf2f9fcddbe6e7be27469db0c9a09366d6fe3f4c90a0bd20b23d5479af0f"}
Nov 03 00:44:48.242868 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.242842    4531 generic.go:356] "Generic (PLEG): container finished" podID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerID="24927a6f3fcbc12665a9f38f12ce895274d79a8145e9fe53b82fef386c94c9bf" exitCode=0
Nov 03 00:44:48.243000 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.242924    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb" event={"ID":"2fea957c-d8e8-4ba8-9564-490b26a120e0","Type":"ContainerDied","Data":"24927a6f3fcbc12665a9f38f12ce895274d79a8145e9fe53b82fef386c94c9bf"}
Nov 03 00:44:48.244938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.244916    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerID="23c064d54ecf4e7375bd5903c90ed27456cd588f93c61672b1acbef05d5b8b48" exitCode=0
Nov 03 00:44:48.245035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.245000    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerDied","Data":"23c064d54ecf4e7375bd5903c90ed27456cd588f93c61672b1acbef05d5b8b48"}
Nov 03 00:44:48.247006 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.246986    4531 generic.go:356] "Generic (PLEG): container finished" podID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerID="fa16e219c936f82b746dd57790ca0f44ed4d013d1371347c0561047d5d4162dc" exitCode=0
Nov 03 00:44:48.247097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.247005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj" event={"ID":"d90cdbb9-26f1-493c-b2fe-7727021e903f","Type":"ContainerDied","Data":"fa16e219c936f82b746dd57790ca0f44ed4d013d1371347c0561047d5d4162dc"}
Nov 03 00:44:48.248950 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.248932    4531 generic.go:356] "Generic (PLEG): container finished" podID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerID="00457ee5617035832a9d38af6305002a8a14998afb81d6f781d97f1fd1b518d3" exitCode=0
Nov 03 00:44:48.249034 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.248999    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw" event={"ID":"daef1caf-7208-4282-b16e-ac5c24306dd1","Type":"ContainerDied","Data":"00457ee5617035832a9d38af6305002a8a14998afb81d6f781d97f1fd1b518d3"}
Nov 03 00:44:48.423270 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.423249    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:48.430455 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.430430    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:48.583469 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583435    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\" (UniqueName: \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\") pod \"61f72592-4afa-4c81-b5c8-871ee26826f9\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") "
Nov 03 00:44:48.583651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\") pod \"61f72592-4afa-4c81-b5c8-871ee26826f9\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") "
Nov 03 00:44:48.583651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583534    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\") pod \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") "
Nov 03 00:44:48.583651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\") pod \"61f72592-4afa-4c81-b5c8-871ee26826f9\" (UID: \"61f72592-4afa-4c81-b5c8-871ee26826f9\") "
Nov 03 00:44:48.583651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583590    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\") pod \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") "
Nov 03 00:44:48.583651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583629    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\" (UniqueName: \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\") pod \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\" (UID: \"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74\") "
Nov 03 00:44:48.584035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.583998    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle" pod "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" (UID: "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:48.584307 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.584285    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle" pod "61f72592-4afa-4c81-b5c8-871ee26826f9" (UID: "61f72592-4afa-4c81-b5c8-871ee26826f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:48.586109 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.586085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv" pod "61f72592-4afa-4c81-b5c8-871ee26826f9" (UID: "61f72592-4afa-4c81-b5c8-871ee26826f9"). InnerVolumeSpecName "kube-api-access-4t2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:48.586109 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.586095    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx" pod "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" (UID: "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74"). InnerVolumeSpecName "kube-api-access-zq4vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:48.588926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.588904    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util" pod "61f72592-4afa-4c81-b5c8-871ee26826f9" (UID: "61f72592-4afa-4c81-b5c8-871ee26826f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:48.589716 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.589697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util" pod "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" (UID: "63ccdda8-f3e3-4a8d-a317-30e2aaa94c74"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:48.684363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684328    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:48.684363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684357    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4vx\" (UniqueName: \"kubernetes.io/projected/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-kube-api-access-zq4vx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:48.684363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684368    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2sv\" (UniqueName: \"kubernetes.io/projected/61f72592-4afa-4c81-b5c8-871ee26826f9-kube-api-access-4t2sv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:48.684587 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684377    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:48.684587 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684387    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63ccdda8-f3e3-4a8d-a317-30e2aaa94c74-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:48.684587 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:48.684396    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61f72592-4afa-4c81-b5c8-871ee26826f9-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.254955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.254863    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt" event={"ID":"61f72592-4afa-4c81-b5c8-871ee26826f9","Type":"ContainerDied","Data":"1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348"}
Nov 03 00:44:49.254955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.254923    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1846db6634a6faf06acb94db8b5737150523f5cd4b86d46252b290a560fc6348"
Nov 03 00:44:49.254955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.254877    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctltmt"
Nov 03 00:44:49.256775 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.256753    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl"
Nov 03 00:44:49.256907 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.256793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m48hl" event={"ID":"63ccdda8-f3e3-4a8d-a317-30e2aaa94c74","Type":"ContainerDied","Data":"15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97"}
Nov 03 00:44:49.256907 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.256838    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15397c2b0f3e17f494220efd34e8b96170c37c831aa2ad1832c6e50470efcf97"
Nov 03 00:44:49.259118 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.259095    4531 generic.go:356] "Generic (PLEG): container finished" podID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerID="b0beb1ef86794e4c3168dd77a2aa57a2c9f102f6fecdb0a881c8970a0cdf6a9e" exitCode=0
Nov 03 00:44:49.259235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.259117    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw" event={"ID":"7df50a03-8c4e-4062-9423-b7834fc1499c","Type":"ContainerDied","Data":"b0beb1ef86794e4c3168dd77a2aa57a2c9f102f6fecdb0a881c8970a0cdf6a9e"}
Nov 03 00:44:49.430187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.427143    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:49.470897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.470876    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:49.473951 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.473929    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:49.477508 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.477477    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:49.480605 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.480588    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:49.491503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.491480    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\") pod \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") "
Nov 03 00:44:49.491644 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.491567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\") pod \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") "
Nov 03 00:44:49.491644 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.491631    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\" (UniqueName: \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\") pod \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\" (UID: \"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33\") "
Nov 03 00:44:49.492493 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.492466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle" pod "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" (UID: "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.494249 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.494225    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d" pod "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" (UID: "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33"). InnerVolumeSpecName "kube-api-access-cfs7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:49.497663 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.497636    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util" pod "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" (UID: "6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.592396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592362    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\") pod \"daef1caf-7208-4282-b16e-ac5c24306dd1\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592416    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\") pod \"2fea957c-d8e8-4ba8-9564-490b26a120e0\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592441    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\") pod \"d90cdbb9-26f1-493c-b2fe-7727021e903f\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592468    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\" (UniqueName: \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\") pod \"daef1caf-7208-4282-b16e-ac5c24306dd1\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\") pod \"2fea957c-d8e8-4ba8-9564-490b26a120e0\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592529    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\" (UniqueName: \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\") pod \"d90cdbb9-26f1-493c-b2fe-7727021e903f\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592554    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\" (UniqueName: \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\") pod \"2fea957c-d8e8-4ba8-9564-490b26a120e0\" (UID: \"2fea957c-d8e8-4ba8-9564-490b26a120e0\") "
Nov 03 00:44:49.592584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592581    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\") pod \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") "
Nov 03 00:44:49.592926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592653    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\") pod \"d90cdbb9-26f1-493c-b2fe-7727021e903f\" (UID: \"d90cdbb9-26f1-493c-b2fe-7727021e903f\") "
Nov 03 00:44:49.592926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592688    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\" (UniqueName: \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\") pod \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") "
Nov 03 00:44:49.592926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592755    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\") pod \"daef1caf-7208-4282-b16e-ac5c24306dd1\" (UID: \"daef1caf-7208-4282-b16e-ac5c24306dd1\") "
Nov 03 00:44:49.592926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.592811    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\") pod \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\" (UID: \"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa\") "
Nov 03 00:44:49.593108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.593064    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.593108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.593083    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.593108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.593099    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfs7d\" (UniqueName: \"kubernetes.io/projected/6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33-kube-api-access-cfs7d\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.593723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.593695    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle" pod "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" (UID: "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.594457 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.594433    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle" pod "d90cdbb9-26f1-493c-b2fe-7727021e903f" (UID: "d90cdbb9-26f1-493c-b2fe-7727021e903f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.595298 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.595269    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle" pod "2fea957c-d8e8-4ba8-9564-490b26a120e0" (UID: "2fea957c-d8e8-4ba8-9564-490b26a120e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.595447 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.595409    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk" pod "d90cdbb9-26f1-493c-b2fe-7727021e903f" (UID: "d90cdbb9-26f1-493c-b2fe-7727021e903f"). InnerVolumeSpecName "kube-api-access-lzgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:49.595756 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.595720    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs" pod "daef1caf-7208-4282-b16e-ac5c24306dd1" (UID: "daef1caf-7208-4282-b16e-ac5c24306dd1"). InnerVolumeSpecName "kube-api-access-kvsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:49.595848 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.595760    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle" pod "daef1caf-7208-4282-b16e-ac5c24306dd1" (UID: "daef1caf-7208-4282-b16e-ac5c24306dd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.596481 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.596462    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d" pod "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" (UID: "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa"). InnerVolumeSpecName "kube-api-access-bg56d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:49.597080 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.597064    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd" pod "2fea957c-d8e8-4ba8-9564-490b26a120e0" (UID: "2fea957c-d8e8-4ba8-9564-490b26a120e0"). InnerVolumeSpecName "kube-api-access-ldjnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:49.599172 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.599134    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util" pod "daef1caf-7208-4282-b16e-ac5c24306dd1" (UID: "daef1caf-7208-4282-b16e-ac5c24306dd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.600195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.600153    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util" pod "2fea957c-d8e8-4ba8-9564-490b26a120e0" (UID: "2fea957c-d8e8-4ba8-9564-490b26a120e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.600752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.600731    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util" pod "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" (UID: "6054ebb6-b128-4a18-9a5e-c6d41f2caeaa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.601035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.601015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util" pod "d90cdbb9-26f1-493c-b2fe-7727021e903f" (UID: "d90cdbb9-26f1-493c-b2fe-7727021e903f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:49.693635 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693599    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693635 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693632    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg56d\" (UniqueName: \"kubernetes.io/projected/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-kube-api-access-bg56d\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693635 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693643    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693652    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693660    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/daef1caf-7208-4282-b16e-ac5c24306dd1-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693668    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693675    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d90cdbb9-26f1-493c-b2fe-7727021e903f-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693683    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvsgs\" (UniqueName: \"kubernetes.io/projected/daef1caf-7208-4282-b16e-ac5c24306dd1-kube-api-access-kvsgs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693691    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fea957c-d8e8-4ba8-9564-490b26a120e0-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693699    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgtk\" (UniqueName: \"kubernetes.io/projected/d90cdbb9-26f1-493c-b2fe-7727021e903f-kube-api-access-lzgtk\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693709    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldjnd\" (UniqueName: \"kubernetes.io/projected/2fea957c-d8e8-4ba8-9564-490b26a120e0-kube-api-access-ldjnd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:49.693845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:49.693717    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6054ebb6-b128-4a18-9a5e-c6d41f2caeaa-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:50.265404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.265363    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj" event={"ID":"d90cdbb9-26f1-493c-b2fe-7727021e903f","Type":"ContainerDied","Data":"aa6e1eaf6adb4c83b173c0560c9c56eec97f63e0fef80efe292e165435141768"}
Nov 03 00:44:50.265811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.265420    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6e1eaf6adb4c83b173c0560c9c56eec97f63e0fef80efe292e165435141768"
Nov 03 00:44:50.265811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.265380    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847answbj"
Nov 03 00:44:50.267400 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.267379    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw"
Nov 03 00:44:50.267400 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.267389    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tgkvw" event={"ID":"daef1caf-7208-4282-b16e-ac5c24306dd1","Type":"ContainerDied","Data":"c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c"}
Nov 03 00:44:50.267549 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.267434    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c13f9811eaf82ba5880f7411f737970f30a70886f375e5bcb9f93078d48b030c"
Nov 03 00:44:50.269480 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.269464    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk"
Nov 03 00:44:50.269596 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.269489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxbsrk" event={"ID":"6054ebb6-b128-4a18-9a5e-c6d41f2caeaa","Type":"ContainerDied","Data":"ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0"}
Nov 03 00:44:50.269596 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.269523    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffdf171185dc331496583b2f4dc98a175782a0b4bec4f11e9e6144c48968c4f0"
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.271625    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb" event={"ID":"2fea957c-d8e8-4ba8-9564-490b26a120e0","Type":"ContainerDied","Data":"3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168"}
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.271642    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7lmzb"
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.271666    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a0878142753770f4f545eec279b4113660fdbf9313ed6192d4d195945bf7168"
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.273602    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th" event={"ID":"6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33","Type":"ContainerDied","Data":"452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225"}
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.273633    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pd5th"
Nov 03 00:44:50.275205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.273640    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="452739db378818450e0b99d0a4e1c6b1044e34b2495be74e9714766ab6615225"
Nov 03 00:44:50.399133 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.399111    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:50.500280 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.500246    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\") pod \"7df50a03-8c4e-4062-9423-b7834fc1499c\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") "
Nov 03 00:44:50.500458 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.500293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\") pod \"7df50a03-8c4e-4062-9423-b7834fc1499c\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") "
Nov 03 00:44:50.500458 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.500333    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\" (UniqueName: \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\") pod \"7df50a03-8c4e-4062-9423-b7834fc1499c\" (UID: \"7df50a03-8c4e-4062-9423-b7834fc1499c\") "
Nov 03 00:44:50.500696 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.500673    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle" pod "7df50a03-8c4e-4062-9423-b7834fc1499c" (UID: "7df50a03-8c4e-4062-9423-b7834fc1499c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:50.502375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.502354    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt" pod "7df50a03-8c4e-4062-9423-b7834fc1499c" (UID: "7df50a03-8c4e-4062-9423-b7834fc1499c"). InnerVolumeSpecName "kube-api-access-cvsrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:44:50.505720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.505701    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util" pod "7df50a03-8c4e-4062-9423-b7834fc1499c" (UID: "7df50a03-8c4e-4062-9423-b7834fc1499c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:44:50.601623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.601548    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-util\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:50.601623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.601575    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsrt\" (UniqueName: \"kubernetes.io/projected/7df50a03-8c4e-4062-9423-b7834fc1499c-kube-api-access-cvsrt\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:50.601623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:50.601586    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7df50a03-8c4e-4062-9423-b7834fc1499c-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:44:51.047887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.047852    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"]
Nov 03 00:44:51.048434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048414    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048436    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048451    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048476    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048487    4531 cpu_manager.go:414] "containerMap: removing container" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048495    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048503    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048509    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048522    4531 cpu_manager.go:414] "containerMap: removing container" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048528    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048537    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048544    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="pull"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048558    4531 cpu_manager.go:414] "containerMap: removing container" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048564    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="util"
Nov 03 00:44:51.048568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048574    4531 cpu_manager.go:414] "containerMap: removing container" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048581    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048590    4531 cpu_manager.go:414] "containerMap: removing container" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048596    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048606    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048613    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048625    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048631    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048641    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048647    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048660    4531 cpu_manager.go:414] "containerMap: removing container" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048666    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048674    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048681    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048689    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048695    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048707    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048745    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048754    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048760    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048768    4531 cpu_manager.go:414] "containerMap: removing container" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048774    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048788    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048794    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048804    4531 cpu_manager.go:414] "containerMap: removing container" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048810    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048820    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048826    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048835    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048842    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="util"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048853    4531 cpu_manager.go:414] "containerMap: removing container" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048859    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048868    4531 cpu_manager.go:414] "containerMap: removing container" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048874    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="pull"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048882    4531 cpu_manager.go:414] "containerMap: removing container" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048888    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048897    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048904    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048915    4531 cpu_manager.go:414] "containerMap: removing container" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="extract"
Nov 03 00:44:51.049235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.048921    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049541    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fea957c-d8e8-4ba8-9564-490b26a120e0" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049564    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6054ebb6-b128-4a18-9a5e-c6d41f2caeaa" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049578    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ccdda8-f3e3-4a8d-a317-30e2aaa94c74" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049585    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab9c75c-be7d-4dfd-88e5-d4cc1af42b33" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049597    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="daef1caf-7208-4282-b16e-ac5c24306dd1" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049604    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90cdbb9-26f1-493c-b2fe-7727021e903f" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049611    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="460c8add-bf5e-41d0-a25d-ed561713040f" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049626    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df50a03-8c4e-4062-9423-b7834fc1499c" containerName="extract"
Nov 03 00:44:51.050818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.049631    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f72592-4afa-4c81-b5c8-871ee26826f9" containerName="extract"
Nov 03 00:44:51.073559 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.073526    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.084433 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.084411    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 00:44:51.086560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.086541    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-p7qvx\""
Nov 03 00:44:51.087396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.087378    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 00:44:51.088915 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.088902    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 00:44:51.089570 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.089557    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 00:44:51.093484 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.093468    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 00:44:51.109235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.109206    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"]
Nov 03 00:44:51.207662 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zz6\" (UniqueName: \"kubernetes.io/projected/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-kube-api-access-88zz6\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.207797 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207662    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.207797 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207705    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.207890 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207802    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.207890 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207853    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.207973 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.207928    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.279869 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.279839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw" event={"ID":"7df50a03-8c4e-4062-9423-b7834fc1499c","Type":"ContainerDied","Data":"2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22"}
Nov 03 00:44:51.280248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.279886    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c19dae0515c6957d027c848f165eba35cf649dd89633266c53b3ee2a6a59d22"
Nov 03 00:44:51.280248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.279889    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fq6jw"
Nov 03 00:44:51.309328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309296    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zz6\" (UniqueName: \"kubernetes.io/projected/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-kube-api-access-88zz6\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309332    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309355    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309374    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309459    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.309861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.309830    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-manifests\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.311870 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.311844    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-tmp\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.311974 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.311945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.312012 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.311989    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.312048 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.312020    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.321336 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.321316    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zz6\" (UniqueName: \"kubernetes.io/projected/a3ce0f76-dc50-4540-9b0f-7daccff99c3d-kube-api-access-88zz6\") pod \"opendatahub-operator-controller-manager-7cc9488979-9nqbf\" (UID: \"a3ce0f76-dc50-4540-9b0f-7daccff99c3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.387547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.387515    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:44:51.682524 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:51.682500    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"]
Nov 03 00:44:51.684836 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:51.684809    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce0f76_dc50_4540_9b0f_7daccff99c3d.slice/crio-de739be75b6dbc48e7b2bc21ba7ac57226e2a0978f745e5fc85f77c5d7c8cce9 WatchSource:0}: Error finding container de739be75b6dbc48e7b2bc21ba7ac57226e2a0978f745e5fc85f77c5d7c8cce9: Status 404 returned error can't find the container with id de739be75b6dbc48e7b2bc21ba7ac57226e2a0978f745e5fc85f77c5d7c8cce9
Nov 03 00:44:52.286571 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:52.286532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf" event={"ID":"a3ce0f76-dc50-4540-9b0f-7daccff99c3d","Type":"ContainerStarted","Data":"de739be75b6dbc48e7b2bc21ba7ac57226e2a0978f745e5fc85f77c5d7c8cce9"}
Nov 03 00:44:57.327860 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.327826    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"]
Nov 03 00:44:57.331280 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.331263    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.346345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.346320    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 00:44:57.348261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.348230    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"]
Nov 03 00:44:57.351459 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.351439    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:44:57.351572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.351555    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7pvc8\""
Nov 03 00:44:57.462243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.462194    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed4ee140-6180-486b-a608-0ff5f28d2355-tmp\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.462420 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.462317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494jq\" (UniqueName: \"kubernetes.io/projected/ed4ee140-6180-486b-a608-0ff5f28d2355-kube-api-access-494jq\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.563487 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.563449    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed4ee140-6180-486b-a608-0ff5f28d2355-tmp\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.563626 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.563508    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494jq\" (UniqueName: \"kubernetes.io/projected/ed4ee140-6180-486b-a608-0ff5f28d2355-kube-api-access-494jq\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.563855 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.563836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed4ee140-6180-486b-a608-0ff5f28d2355-tmp\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.581224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.581177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-494jq\" (UniqueName: \"kubernetes.io/projected/ed4ee140-6180-486b-a608-0ff5f28d2355-kube-api-access-494jq\") pod \"jobset-operator-674cd59dc5-s6gsn\" (UID: \"ed4ee140-6180-486b-a608-0ff5f28d2355\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.664865 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.664833    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"
Nov 03 00:44:57.970190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:57.970088    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn"]
Nov 03 00:44:57.973110 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:44:57.973082    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded4ee140_6180_486b_a608_0ff5f28d2355.slice/crio-c06cfb7d2c12b97ab586f33ca63110553eb794d9421815b45e969dafb1b39ae4 WatchSource:0}: Error finding container c06cfb7d2c12b97ab586f33ca63110553eb794d9421815b45e969dafb1b39ae4: Status 404 returned error can't find the container with id c06cfb7d2c12b97ab586f33ca63110553eb794d9421815b45e969dafb1b39ae4
Nov 03 00:44:58.316780 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:44:58.316688    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn" event={"ID":"ed4ee140-6180-486b-a608-0ff5f28d2355","Type":"ContainerStarted","Data":"c06cfb7d2c12b97ab586f33ca63110553eb794d9421815b45e969dafb1b39ae4"}
Nov 03 00:45:00.337979 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:00.337944    4531 generic.go:356] "Generic (PLEG): container finished" podID="a3ce0f76-dc50-4540-9b0f-7daccff99c3d" containerID="f91c2ed60cc599b39c289268fd92197696da61e8f463ce55006bc9f90524ed56" exitCode=0
Nov 03 00:45:00.338464 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:00.338027    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf" event={"ID":"a3ce0f76-dc50-4540-9b0f-7daccff99c3d","Type":"ContainerDied","Data":"f91c2ed60cc599b39c289268fd92197696da61e8f463ce55006bc9f90524ed56"}
Nov 03 00:45:01.343710 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:01.343675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn" event={"ID":"ed4ee140-6180-486b-a608-0ff5f28d2355","Type":"ContainerStarted","Data":"8ad77178d55064971a52f5061d3cdc1bf8394da9b36abf7d69639b5224bb9cd3"}
Nov 03 00:45:01.345565 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:01.345541    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf" event={"ID":"a3ce0f76-dc50-4540-9b0f-7daccff99c3d","Type":"ContainerStarted","Data":"ffd897a0237194ef894e835f003bf029ee4a130b8c3ac3e26c7f634189f9ae56"}
Nov 03 00:45:01.345691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:01.345680    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:45:01.360746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:01.360702    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-s6gsn" podStartSLOduration=1.958803772 podStartE2EDuration="4.360689207s" totalImagesPullingTime="2.401885435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.359320769 +0000 UTC m=+246.573064443" watchObservedRunningTime="2026-11-03 00:45:01.360689207 +0000 UTC m=+246.574432879"
Nov 03 00:45:01.394083 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:01.394031    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf" podStartSLOduration=1.506453083 podStartE2EDuration="10.39401426s" totalImagesPullingTime="8.354535215s" totalInitContainerRuntime="533.025962ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.392231621 +0000 UTC m=+246.605975305" watchObservedRunningTime="2026-11-03 00:45:01.39401426 +0000 UTC m=+246.607757938"
Nov 03 00:45:08.327099 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.327057    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"]
Nov 03 00:45:08.330352 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.330335    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.339781 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.339756    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:45:08.340872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.340837    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m8rvk\""
Nov 03 00:45:08.340872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.340862    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 00:45:08.343496 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.343474    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"]
Nov 03 00:45:08.459112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.459076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzbc\" (UniqueName: \"kubernetes.io/projected/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-kube-api-access-8hzbc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.459300 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.459125    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.560077 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.560043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzbc\" (UniqueName: \"kubernetes.io/projected/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-kube-api-access-8hzbc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.560260 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.560091    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.560505 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.560485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.568224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.568202    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzbc\" (UniqueName: \"kubernetes.io/projected/4f1c09f5-3e09-4609-807c-5d7f774d8a8f-kube-api-access-8hzbc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd\" (UID: \"4f1c09f5-3e09-4609-807c-5d7f774d8a8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.641724 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.641652    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"
Nov 03 00:45:08.923813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:08.923745    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd"]
Nov 03 00:45:08.927818 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:08.927791    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1c09f5_3e09_4609_807c_5d7f774d8a8f.slice/crio-ec24c52d110394a88bc23d3de1cd791e44c1dad48dd4c29f678fcf80ffcf1e26 WatchSource:0}: Error finding container ec24c52d110394a88bc23d3de1cd791e44c1dad48dd4c29f678fcf80ffcf1e26: Status 404 returned error can't find the container with id ec24c52d110394a88bc23d3de1cd791e44c1dad48dd4c29f678fcf80ffcf1e26
Nov 03 00:45:09.382154 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:09.382125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd" event={"ID":"4f1c09f5-3e09-4609-807c-5d7f774d8a8f","Type":"ContainerStarted","Data":"ec24c52d110394a88bc23d3de1cd791e44c1dad48dd4c29f678fcf80ffcf1e26"}
Nov 03 00:45:09.876020 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:45:09.875984    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:45:11.341558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.341516    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:11.356225 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.356188    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.357540 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.357511    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:11.360392 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.360368    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2mtm8\""
Nov 03 00:45:11.360524 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.360383    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 00:45:11.361334 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.361317    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 00:45:11.485854 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.485815    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.486066 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.485983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjmt\" (UniqueName: \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.586574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.586536    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjmt\" (UniqueName: \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.586574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.586580    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.586873 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.586857    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.597301 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.597244    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjmt\" (UniqueName: \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f5v7z\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:11.687251 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:11.687225    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:12.007527 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:12.007496    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:12.008947 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:12.008915    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9430b02_8fb4_47ee_a59e_bb5951578cdd.slice/crio-4634b4b33cc2336bfb6076f88e01325b5523191e5dc808ba26aca08ca462d00b WatchSource:0}: Error finding container 4634b4b33cc2336bfb6076f88e01325b5523191e5dc808ba26aca08ca462d00b: Status 404 returned error can't find the container with id 4634b4b33cc2336bfb6076f88e01325b5523191e5dc808ba26aca08ca462d00b
Nov 03 00:45:12.352034 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:12.352005    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7cc9488979-9nqbf"
Nov 03 00:45:12.400846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:12.400810    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd" event={"ID":"4f1c09f5-3e09-4609-807c-5d7f774d8a8f","Type":"ContainerStarted","Data":"38619fc782196355a315e1d8689cecbf5c4da7b4bb5ffb98f183d1cb20087779"}
Nov 03 00:45:12.402572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:12.402521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" event={"ID":"a9430b02-8fb4-47ee-a59e-bb5951578cdd","Type":"ContainerStarted","Data":"4634b4b33cc2336bfb6076f88e01325b5523191e5dc808ba26aca08ca462d00b"}
Nov 03 00:45:12.443935 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:12.443858    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bhtmd" podStartSLOduration=1.712570618 podStartE2EDuration="4.443838577s" totalImagesPullingTime="2.731267959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:12.442085497 +0000 UTC m=+257.655829170" watchObservedRunningTime="2026-11-03 00:45:12.443838577 +0000 UTC m=+257.657582250"
Nov 03 00:45:15.100098 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.100063    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"]
Nov 03 00:45:15.104028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.104006    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:15.115111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.115076    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hxglz\""
Nov 03 00:45:15.117516 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.117449    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"]
Nov 03 00:45:15.117663 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.117623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgv4\" (UniqueName: \"kubernetes.io/projected/d65cb9f3-5969-41c3-8f50-42bf0f002c72-kube-api-access-pvgv4\") pod \"authorino-operator-7b7bb4bc77-c9svn\" (UID: \"d65cb9f3-5969-41c3-8f50-42bf0f002c72\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:15.219005 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.218965    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgv4\" (UniqueName: \"kubernetes.io/projected/d65cb9f3-5969-41c3-8f50-42bf0f002c72-kube-api-access-pvgv4\") pod \"authorino-operator-7b7bb4bc77-c9svn\" (UID: \"d65cb9f3-5969-41c3-8f50-42bf0f002c72\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:15.244371 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.244304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgv4\" (UniqueName: \"kubernetes.io/projected/d65cb9f3-5969-41c3-8f50-42bf0f002c72-kube-api-access-pvgv4\") pod \"authorino-operator-7b7bb4bc77-c9svn\" (UID: \"d65cb9f3-5969-41c3-8f50-42bf0f002c72\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:15.421473 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.421200    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:15.723886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:15.723858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"]
Nov 03 00:45:15.726107 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:15.726073    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65cb9f3_5969_41c3_8f50_42bf0f002c72.slice/crio-2024ede921fcc64fc10cf8e58d55cf39d661d0c55359222b68a892516ecb4354 WatchSource:0}: Error finding container 2024ede921fcc64fc10cf8e58d55cf39d661d0c55359222b68a892516ecb4354: Status 404 returned error can't find the container with id 2024ede921fcc64fc10cf8e58d55cf39d661d0c55359222b68a892516ecb4354
Nov 03 00:45:16.433454 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:16.433408    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn" event={"ID":"d65cb9f3-5969-41c3-8f50-42bf0f002c72","Type":"ContainerStarted","Data":"2024ede921fcc64fc10cf8e58d55cf39d661d0c55359222b68a892516ecb4354"}
Nov 03 00:45:17.440928 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:17.440899    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" event={"ID":"a9430b02-8fb4-47ee-a59e-bb5951578cdd","Type":"ContainerStarted","Data":"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"}
Nov 03 00:45:17.441312 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:17.440972    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:17.463777 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:17.463731    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" podStartSLOduration=1.816788275 podStartE2EDuration="6.463717552s" totalImagesPullingTime="4.646929277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:17.460894769 +0000 UTC m=+262.674638441" watchObservedRunningTime="2026-11-03 00:45:17.463717552 +0000 UTC m=+262.677461223"
Nov 03 00:45:18.446898 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:18.446854    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn" event={"ID":"d65cb9f3-5969-41c3-8f50-42bf0f002c72","Type":"ContainerStarted","Data":"0d7fad421a08117fc0484bc3e98703ebbd6519773b1edc3988d1658c55858b9e"}
Nov 03 00:45:18.447314 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:18.446987    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:18.466032 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:18.465981    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn" podStartSLOduration=1.762740931 podStartE2EDuration="3.46596903s" totalImagesPullingTime="1.703228099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:18.464221286 +0000 UTC m=+263.677964959" watchObservedRunningTime="2026-11-03 00:45:18.46596903 +0000 UTC m=+263.679712698"
Nov 03 00:45:19.192566 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.192534    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"]
Nov 03 00:45:19.196241 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.196218    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:19.206925 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.206903    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"]
Nov 03 00:45:19.250665 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.250631    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wxd\" (UniqueName: \"kubernetes.io/projected/dbf70c07-8ae7-4884-933c-17d7dccf6806-kube-api-access-27wxd\") pod \"limitador-operator-controller-manager-5778bb4449-szqjl\" (UID: \"dbf70c07-8ae7-4884-933c-17d7dccf6806\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:19.270780 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.270756    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pcq96\""
Nov 03 00:45:19.351647 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.351610    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wxd\" (UniqueName: \"kubernetes.io/projected/dbf70c07-8ae7-4884-933c-17d7dccf6806-kube-api-access-27wxd\") pod \"limitador-operator-controller-manager-5778bb4449-szqjl\" (UID: \"dbf70c07-8ae7-4884-933c-17d7dccf6806\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:19.363558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.363528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wxd\" (UniqueName: \"kubernetes.io/projected/dbf70c07-8ae7-4884-933c-17d7dccf6806-kube-api-access-27wxd\") pod \"limitador-operator-controller-manager-5778bb4449-szqjl\" (UID: \"dbf70c07-8ae7-4884-933c-17d7dccf6806\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:19.509644 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.509559    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:19.788299 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:19.788268    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"]
Nov 03 00:45:19.790760 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:19.790730    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf70c07_8ae7_4884_933c_17d7dccf6806.slice/crio-f41720d771ea94c39ae005026de169cdf0e8166074a23ef4c1ad50aabd93f516 WatchSource:0}: Error finding container f41720d771ea94c39ae005026de169cdf0e8166074a23ef4c1ad50aabd93f516: Status 404 returned error can't find the container with id f41720d771ea94c39ae005026de169cdf0e8166074a23ef4c1ad50aabd93f516
Nov 03 00:45:20.456534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:20.456498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl" event={"ID":"dbf70c07-8ae7-4884-933c-17d7dccf6806","Type":"ContainerStarted","Data":"f41720d771ea94c39ae005026de169cdf0e8166074a23ef4c1ad50aabd93f516"}
Nov 03 00:45:22.467096 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:22.467060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl" event={"ID":"dbf70c07-8ae7-4884-933c-17d7dccf6806","Type":"ContainerStarted","Data":"d408365f9d2789c3dafbadf39eccb4cbba793db7e6c8c5cf3ac0a15f7577b428"}
Nov 03 00:45:22.467565 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:22.467184    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:22.485355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:22.485305    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl" podStartSLOduration=1.270976611 podStartE2EDuration="3.485292813s" totalImagesPullingTime="2.214316202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:22.484376548 +0000 UTC m=+267.698120221" watchObservedRunningTime="2026-11-03 00:45:22.485292813 +0000 UTC m=+267.699036485"
Nov 03 00:45:23.018457 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.018423    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"]
Nov 03 00:45:23.022257 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.022228    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:23.031325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.031298    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 00:45:23.031325 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.031298    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tkt9s\""
Nov 03 00:45:23.032846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.032826    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"]
Nov 03 00:45:23.084666 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.084636    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8vk\" (UniqueName: \"kubernetes.io/projected/df529571-d827-4799-9b09-3e0ec8b4ce56-kube-api-access-ds8vk\") pod \"dns-operator-controller-manager-8557cbdb84-sw79f\" (UID: \"df529571-d827-4799-9b09-3e0ec8b4ce56\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:23.185515 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.185479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8vk\" (UniqueName: \"kubernetes.io/projected/df529571-d827-4799-9b09-3e0ec8b4ce56-kube-api-access-ds8vk\") pod \"dns-operator-controller-manager-8557cbdb84-sw79f\" (UID: \"df529571-d827-4799-9b09-3e0ec8b4ce56\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:23.197255 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.197221    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8vk\" (UniqueName: \"kubernetes.io/projected/df529571-d827-4799-9b09-3e0ec8b4ce56-kube-api-access-ds8vk\") pod \"dns-operator-controller-manager-8557cbdb84-sw79f\" (UID: \"df529571-d827-4799-9b09-3e0ec8b4ce56\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:23.335786 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.335754    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:23.610934 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:23.610880    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"]
Nov 03 00:45:23.613118 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:23.613090    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf529571_d827_4799_9b09_3e0ec8b4ce56.slice/crio-2168cae79d3a332c4d5e89ebefe21c27bcd4bd6b6e8efaefb9c155d4f3393aef WatchSource:0}: Error finding container 2168cae79d3a332c4d5e89ebefe21c27bcd4bd6b6e8efaefb9c155d4f3393aef: Status 404 returned error can't find the container with id 2168cae79d3a332c4d5e89ebefe21c27bcd4bd6b6e8efaefb9c155d4f3393aef
Nov 03 00:45:24.478003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:24.477964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" event={"ID":"df529571-d827-4799-9b09-3e0ec8b4ce56","Type":"ContainerStarted","Data":"2168cae79d3a332c4d5e89ebefe21c27bcd4bd6b6e8efaefb9c155d4f3393aef"}
Nov 03 00:45:26.489262 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:26.489222    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" event={"ID":"df529571-d827-4799-9b09-3e0ec8b4ce56","Type":"ContainerStarted","Data":"c48db5dca8a3ed3891e64f2d686edef2508c777256b0d86e0c9ebdda7dd21a1e"}
Nov 03 00:45:26.489657 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:26.489318    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:26.510258 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:26.510208    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" podStartSLOduration=2.213015801 podStartE2EDuration="4.510193738s" totalImagesPullingTime="2.297177937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:26.507941531 +0000 UTC m=+271.721685204" watchObservedRunningTime="2026-11-03 00:45:26.510193738 +0000 UTC m=+271.723937410"
Nov 03 00:45:28.449081 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:28.449045    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:28.948289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:28.948251    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 00:45:28.951799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:28.951783    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:28.971055 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:28.971028    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 00:45:29.036535 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.036503    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.036693 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.036585    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlrn\" (UniqueName: \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.137815 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.137780    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.137982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.137824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlrn\" (UniqueName: \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.138187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.138140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.150194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.150150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlrn\" (UniqueName: \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\") pod \"kuadrant-operator-controller-manager-678d87766f-7zsk9\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.264991 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.264905    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:29.453782 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.453752    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c9svn"
Nov 03 00:45:29.543926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:29.543899    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 00:45:29.545775 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:29.545741    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d9bc3e_fee0_499f_9c05_f6fcdd169cfd.slice/crio-775105cf7e2758b842c64c37a856296613f7af4812999e2ad0db03666f351af8 WatchSource:0}: Error finding container 775105cf7e2758b842c64c37a856296613f7af4812999e2ad0db03666f351af8: Status 404 returned error can't find the container with id 775105cf7e2758b842c64c37a856296613f7af4812999e2ad0db03666f351af8
Nov 03 00:45:30.512831 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.512794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" event={"ID":"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd","Type":"ContainerStarted","Data":"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"}
Nov 03 00:45:30.513324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.512843    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" event={"ID":"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd","Type":"ContainerStarted","Data":"775105cf7e2758b842c64c37a856296613f7af4812999e2ad0db03666f351af8"}
Nov 03 00:45:30.513324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.512882    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:30.537146 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.537095    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" podStartSLOduration=2.53708153 podStartE2EDuration="2.53708153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:30.535284058 +0000 UTC m=+275.749027731" watchObservedRunningTime="2026-11-03 00:45:30.53708153 +0000 UTC m=+275.750825201"
Nov 03 00:45:30.911785 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.911755    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"]
Nov 03 00:45:30.915289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.915266    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.923150 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.923128    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-znw7l\""
Nov 03 00:45:30.925338 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.925294    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"]
Nov 03 00:45:30.955878 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.955841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5hm\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-kube-api-access-hp5hm\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.955891    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.955981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a72581-3f3f-4ff5-b75c-c97eba306705-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956030    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956300 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956186    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956300 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:30.956411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:30.956346    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057398 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057414    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057462    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057494    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5hm\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-kube-api-access-hp5hm\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057541    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057638    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057675    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a72581-3f3f-4ff5-b75c-c97eba306705-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057726    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.057971 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.057943    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.058120 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.058098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.058222 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.058126    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.058222 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.058137    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.058413 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.058393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a72581-3f3f-4ff5-b75c-c97eba306705-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.059841 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.059816    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.060011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.059996    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.080723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.080694    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5hm\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-kube-api-access-hp5hm\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.080830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.080704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a72581-3f3f-4ff5-b75c-c97eba306705-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6644c85865gxcxg\" (UID: \"65a72581-3f3f-4ff5-b75c-c97eba306705\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.232410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.232316    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:31.529872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.529845    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"]
Nov 03 00:45:31.532287 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:31.532252    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a72581_3f3f_4ff5_b75c_c97eba306705.slice/crio-bd7903533de61eeb93f0fb4104758d2b46c9bb955224692068573f44f14a62f7 WatchSource:0}: Error finding container bd7903533de61eeb93f0fb4104758d2b46c9bb955224692068573f44f14a62f7: Status 404 returned error can't find the container with id bd7903533de61eeb93f0fb4104758d2b46c9bb955224692068573f44f14a62f7
Nov 03 00:45:31.534384 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.534348    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:31.534463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.534428    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:31.534463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:31.534459    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:32.524933 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.524895    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg" event={"ID":"65a72581-3f3f-4ff5-b75c-c97eba306705","Type":"ContainerStarted","Data":"822786493d598028bbe7e86707ef0d903e0131f18175d486e3f65456c06d3921"}
Nov 03 00:45:32.525132 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.524962    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg" event={"ID":"65a72581-3f3f-4ff5-b75c-c97eba306705","Type":"ContainerStarted","Data":"bd7903533de61eeb93f0fb4104758d2b46c9bb955224692068573f44f14a62f7"}
Nov 03 00:45:32.544329 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.544272    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg" podStartSLOduration=2.544250631 podStartE2EDuration="2.544250631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:32.543477705 +0000 UTC m=+277.757221391" watchObservedRunningTime="2026-11-03 00:45:32.544250631 +0000 UTC m=+277.757994308"
Nov 03 00:45:32.822434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.822386    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"]
Nov 03 00:45:32.826055 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.826034    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.832128 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.832053    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 00:45:32.832297 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.832178    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 00:45:32.832297 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.832274    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-5499b\""
Nov 03 00:45:32.835468 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.835445    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 00:45:32.838076 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.838052    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"]
Nov 03 00:45:32.874052 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.874017    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhdt\" (UniqueName: \"kubernetes.io/projected/29693553-4e1c-4e36-aad0-a1360b84ba72-kube-api-access-4rhdt\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.874266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.874105    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.874266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.874148    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29693553-4e1c-4e36-aad0-a1360b84ba72-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.874266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.874259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975123 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975082    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975138    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhdt\" (UniqueName: \"kubernetes.io/projected/29693553-4e1c-4e36-aad0-a1360b84ba72-kube-api-access-4rhdt\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975355 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975299    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29693553-4e1c-4e36-aad0-a1360b84ba72-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975755    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.975914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.975869    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29693553-4e1c-4e36-aad0-a1360b84ba72-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.976732 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.976714    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29693553-4e1c-4e36-aad0-a1360b84ba72-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:32.983888 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:32.983861    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhdt\" (UniqueName: \"kubernetes.io/projected/29693553-4e1c-4e36-aad0-a1360b84ba72-kube-api-access-4rhdt\") pod \"kserve-module-controller-manager-7566cf5fc8-n922j\" (UID: \"29693553-4e1c-4e36-aad0-a1360b84ba72\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:33.140375 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.140295    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:33.233586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.233552    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:33.238283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.238243    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:33.423463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.423432    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"]
Nov 03 00:45:33.425669 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:33.425644    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29693553_4e1c_4e36_aad0_a1360b84ba72.slice/crio-7a011b03bf55a1b259baea627b2801c7a97d531e73aa262d277d43c368bbb690 WatchSource:0}: Error finding container 7a011b03bf55a1b259baea627b2801c7a97d531e73aa262d277d43c368bbb690: Status 404 returned error can't find the container with id 7a011b03bf55a1b259baea627b2801c7a97d531e73aa262d277d43c368bbb690
Nov 03 00:45:33.475400 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.475370    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-szqjl"
Nov 03 00:45:33.531316 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.531278    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j" event={"ID":"29693553-4e1c-4e36-aad0-a1360b84ba72","Type":"ContainerStarted","Data":"7a011b03bf55a1b259baea627b2801c7a97d531e73aa262d277d43c368bbb690"}
Nov 03 00:45:33.531534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.531515    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:33.532499 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:33.532478    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6644c85865gxcxg"
Nov 03 00:45:36.553704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:36.553662    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j" event={"ID":"29693553-4e1c-4e36-aad0-a1360b84ba72","Type":"ContainerStarted","Data":"36d0709bd6ecb2fd3e1b06ea5e1fbc5f5e93d3084beed5cdefa110a4b0e8758d"}
Nov 03 00:45:36.554254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:36.553741    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:45:36.574229 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:36.574176    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j" podStartSLOduration=2.025561929 podStartE2EDuration="4.574146873s" totalImagesPullingTime="2.548584944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:36.572496283 +0000 UTC m=+281.786239956" watchObservedRunningTime="2026-11-03 00:45:36.574146873 +0000 UTC m=+281.787890545"
Nov 03 00:45:37.496880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:37.496851    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 00:45:39.875987 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:45:39.875950    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:45:41.520899 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.520863    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 00:45:41.574339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.574306    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:41.574571 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.574545    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" podUID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" containerName="manager" containerID="cri-o://e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98" gracePeriod=10
Nov 03 00:45:41.815379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.815353    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:41.949700 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.949672    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\") pod \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") "
Nov 03 00:45:41.949859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.949721    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\" (UniqueName: \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\") pod \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\" (UID: \"a9430b02-8fb4-47ee-a59e-bb5951578cdd\") "
Nov 03 00:45:41.950184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.950138    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume" pod "a9430b02-8fb4-47ee-a59e-bb5951578cdd" (UID: "a9430b02-8fb4-47ee-a59e-bb5951578cdd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:41.951815 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:41.951789    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt" pod "a9430b02-8fb4-47ee-a59e-bb5951578cdd" (UID: "a9430b02-8fb4-47ee-a59e-bb5951578cdd"). InnerVolumeSpecName "kube-api-access-vwjmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:42.050652 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.050575    4531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9430b02-8fb4-47ee-a59e-bb5951578cdd-extensions-socket-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:45:42.050652 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.050602    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjmt\" (UniqueName: \"kubernetes.io/projected/a9430b02-8fb4-47ee-a59e-bb5951578cdd-kube-api-access-vwjmt\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:45:42.587612 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.587576    4531 generic.go:356] "Generic (PLEG): container finished" podID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" containerID="e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98" exitCode=0
Nov 03 00:45:42.588013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.587618    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"
Nov 03 00:45:42.588013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.587663    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" event={"ID":"a9430b02-8fb4-47ee-a59e-bb5951578cdd","Type":"ContainerDied","Data":"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"}
Nov 03 00:45:42.588013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.587716    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z" event={"ID":"a9430b02-8fb4-47ee-a59e-bb5951578cdd","Type":"ContainerDied","Data":"4634b4b33cc2336bfb6076f88e01325b5523191e5dc808ba26aca08ca462d00b"}
Nov 03 00:45:42.588013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.587737    4531 scope.go:122] "RemoveContainer" containerID="e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"
Nov 03 00:45:42.597604 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.597586    4531 scope.go:122] "RemoveContainer" containerID="e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"
Nov 03 00:45:42.597866 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:45:42.597847    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98\": container with ID starting with e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98 not found: ID does not exist" containerID="e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"
Nov 03 00:45:42.597917 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.597875    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98"} err="failed to get container status \"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98\": rpc error: code = NotFound desc = could not find container \"e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98\": container with ID starting with e3012c9fbe79e12d201f43ebf103c6643bad771ab563fd90d27bce2d3c126b98 not found: ID does not exist"
Nov 03 00:45:42.612730 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.612701    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:42.616723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.616704    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f5v7z"]
Nov 03 00:45:42.901322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:42.901247    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" path="/var/lib/kubelet/pods/a9430b02-8fb4-47ee-a59e-bb5951578cdd/volumes"
Nov 03 00:45:50.757485 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.757452    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:45:50.757943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.757808    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" containerName="manager"
Nov 03 00:45:50.757943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.757818    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" containerName="manager"
Nov 03 00:45:50.757943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.757903    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9430b02-8fb4-47ee-a59e-bb5951578cdd" containerName="manager"
Nov 03 00:45:50.763038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.763019    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.767407 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.767382    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 00:45:50.768392 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.768370    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w9pxr\""
Nov 03 00:45:50.770796 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.770774    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:45:50.826546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.826517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.826706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.826652    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2wf\" (UniqueName: \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.860084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.860052    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:45:50.927110 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.927076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2wf\" (UniqueName: \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.927286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.927130    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.927736 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.927718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:50.935548 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:50.935516    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2wf\" (UniqueName: \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\") pod \"limitador-limitador-74dfcfb894-thg2j\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:51.076547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:51.076521    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:51.353625 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:51.353600    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:45:51.355337 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:45:51.355304    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5216c164_9950_4f86_8c4b_2596c229f56f.slice/crio-ad75f4767c758d7f713b2613d4a13352aba5c430f8a6a480fc96803b6afd2a89 WatchSource:0}: Error finding container ad75f4767c758d7f713b2613d4a13352aba5c430f8a6a480fc96803b6afd2a89: Status 404 returned error can't find the container with id ad75f4767c758d7f713b2613d4a13352aba5c430f8a6a480fc96803b6afd2a89
Nov 03 00:45:51.634071 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:51.633989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" event={"ID":"5216c164-9950-4f86-8c4b-2596c229f56f","Type":"ContainerStarted","Data":"ad75f4767c758d7f713b2613d4a13352aba5c430f8a6a480fc96803b6afd2a89"}
Nov 03 00:45:53.646057 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:53.646022    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" event={"ID":"5216c164-9950-4f86-8c4b-2596c229f56f","Type":"ContainerStarted","Data":"7e80f182f40eb90a0c61f02a0e752a81d8fde3037ce68b8d4126a8aa6b79b9b9"}
Nov 03 00:45:53.646468 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:53.646149    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:45:53.662647 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:53.662562    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" podStartSLOduration=1.617944931 podStartE2EDuration="3.662548093s" totalImagesPullingTime="2.044603162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:53.660900368 +0000 UTC m=+298.874644042" watchObservedRunningTime="2026-11-03 00:45:53.662548093 +0000 UTC m=+298.876291763"
Nov 03 00:45:57.560548 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:45:57.560513    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-n922j"
Nov 03 00:46:04.651964 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:04.651931    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:46:09.875200 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:46:09.875095    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:46:14.201238 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:14.201199    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:46:14.201784 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:14.201400    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" containerName="limitador" containerID="cri-o://7e80f182f40eb90a0c61f02a0e752a81d8fde3037ce68b8d4126a8aa6b79b9b9" gracePeriod=30
Nov 03 00:46:14.651854 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:14.651801    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" containerName="limitador" probeResult="failure" output="Get \"http://10.133.0.41:8080/status\": dial tcp 10.133.0.41:8080: connect: connection refused"
Nov 03 00:46:14.758827 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:14.758792    4531 generic.go:356] "Generic (PLEG): container finished" podID="5216c164-9950-4f86-8c4b-2596c229f56f" containerID="7e80f182f40eb90a0c61f02a0e752a81d8fde3037ce68b8d4126a8aa6b79b9b9" exitCode=0
Nov 03 00:46:14.758975 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:14.758865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" event={"ID":"5216c164-9950-4f86-8c4b-2596c229f56f","Type":"ContainerDied","Data":"7e80f182f40eb90a0c61f02a0e752a81d8fde3037ce68b8d4126a8aa6b79b9b9"}
Nov 03 00:46:15.143272 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.143244    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:46:15.247691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.247653    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\" (UniqueName: \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\") pod \"5216c164-9950-4f86-8c4b-2596c229f56f\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") "
Nov 03 00:46:15.248213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.247721    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\" (UniqueName: \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\") pod \"5216c164-9950-4f86-8c4b-2596c229f56f\" (UID: \"5216c164-9950-4f86-8c4b-2596c229f56f\") "
Nov 03 00:46:15.248213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.248137    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file" pod "5216c164-9950-4f86-8c4b-2596c229f56f" (UID: "5216c164-9950-4f86-8c4b-2596c229f56f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:15.249956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.249925    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf" pod "5216c164-9950-4f86-8c4b-2596c229f56f" (UID: "5216c164-9950-4f86-8c4b-2596c229f56f"). InnerVolumeSpecName "kube-api-access-2g2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:15.349324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.349288    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2wf\" (UniqueName: \"kubernetes.io/projected/5216c164-9950-4f86-8c4b-2596c229f56f-kube-api-access-2g2wf\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:46:15.349324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.349323    4531 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5216c164-9950-4f86-8c4b-2596c229f56f-config-file\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:46:15.764594 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.764499    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j" event={"ID":"5216c164-9950-4f86-8c4b-2596c229f56f","Type":"ContainerDied","Data":"ad75f4767c758d7f713b2613d4a13352aba5c430f8a6a480fc96803b6afd2a89"}
Nov 03 00:46:15.764740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.764521    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-thg2j"
Nov 03 00:46:15.764740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.764619    4531 scope.go:122] "RemoveContainer" containerID="7e80f182f40eb90a0c61f02a0e752a81d8fde3037ce68b8d4126a8aa6b79b9b9"
Nov 03 00:46:15.788481 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.788454    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:46:15.794846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:15.794820    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-thg2j"]
Nov 03 00:46:16.901293 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:16.901256    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" path="/var/lib/kubelet/pods/5216c164-9950-4f86-8c4b-2596c229f56f/volumes"
Nov 03 00:46:19.510937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.510900    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"]
Nov 03 00:46:19.511445 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.511309    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" containerName="limitador"
Nov 03 00:46:19.511445 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.511323    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" containerName="limitador"
Nov 03 00:46:19.511445 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.511411    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5216c164-9950-4f86-8c4b-2596c229f56f" containerName="limitador"
Nov 03 00:46:19.515623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.515606    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.532774 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.532738    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:46:19.532872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.532829    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 00:46:19.532872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.532839    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:46:19.533972 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.533951    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gnfs7\""
Nov 03 00:46:19.536678 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.536658    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"]
Nov 03 00:46:19.587303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.587275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/074e788b-3f4c-4b98-a302-c67107a603bc-data\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.587458 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.587309    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtnl\" (UniqueName: \"kubernetes.io/projected/074e788b-3f4c-4b98-a302-c67107a603bc-kube-api-access-rrtnl\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.690363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.687968    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/074e788b-3f4c-4b98-a302-c67107a603bc-data\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.690544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.690413    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtnl\" (UniqueName: \"kubernetes.io/projected/074e788b-3f4c-4b98-a302-c67107a603bc-kube-api-access-rrtnl\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.690544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.689058    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/074e788b-3f4c-4b98-a302-c67107a603bc-data\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.706863 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.706820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtnl\" (UniqueName: \"kubernetes.io/projected/074e788b-3f4c-4b98-a302-c67107a603bc-kube-api-access-rrtnl\") pod \"postgres-7cdbf6dc44-kt22v\" (UID: \"074e788b-3f4c-4b98-a302-c67107a603bc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:19.828024 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:19.827996    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:20.104801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:20.104731    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"]
Nov 03 00:46:20.106992 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:20.106962    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074e788b_3f4c_4b98_a302_c67107a603bc.slice/crio-e119e1799a5ea3740d1b3ae6f1978142e719ebafa99fec972638fd33450bbedb WatchSource:0}: Error finding container e119e1799a5ea3740d1b3ae6f1978142e719ebafa99fec972638fd33450bbedb: Status 404 returned error can't find the container with id e119e1799a5ea3740d1b3ae6f1978142e719ebafa99fec972638fd33450bbedb
Nov 03 00:46:20.108302 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:20.108282    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:46:20.801838 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:20.801802    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v" event={"ID":"074e788b-3f4c-4b98-a302-c67107a603bc","Type":"ContainerStarted","Data":"e119e1799a5ea3740d1b3ae6f1978142e719ebafa99fec972638fd33450bbedb"}
Nov 03 00:46:26.033294 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.031595    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:46:26.040555 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.040523    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mpgsb"]
Nov 03 00:46:26.040822 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.040624    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.046699 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.046656    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ctxbc"]
Nov 03 00:46:26.048416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.048391    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.055852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.051238    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 00:46:26.055852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.051377    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sb4ch\""
Nov 03 00:46:26.059716 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.059688    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knbqk"]
Nov 03 00:46:26.061101 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.059684    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.063800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.063555    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:46:26.063800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.063594    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mpgsb"]
Nov 03 00:46:26.063800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.063621    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ctxbc"]
Nov 03 00:46:26.063800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.063689    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.064286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.064258    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 00:46:26.064382 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.064350    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qj86n\""
Nov 03 00:46:26.073713 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.073645    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knbqk"]
Nov 03 00:46:26.094210 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.094155    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bmq4x\""
Nov 03 00:46:26.094397 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.094267    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hkpfm\""
Nov 03 00:46:26.094462 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.094397    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 00:46:26.094462 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.094327    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 00:46:26.094462 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.094271    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 00:46:26.159243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcx5\" (UniqueName: \"kubernetes.io/projected/61eb856b-5461-45bb-9126-c9d33d325ae7-kube-api-access-kbcx5\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.159424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159252    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fee83f-2189-40b6-8559-022d6cc91770-tls-certs\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.159424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159376    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-metrics-certs\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.159424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159410    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61eb856b-5461-45bb-9126-c9d33d325ae7-cert\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.159593 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htp4f\" (UniqueName: \"kubernetes.io/projected/11772d42-58e7-41ad-8adb-28220a88c2f9-kube-api-access-htp4f\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.159593 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159506    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.159694 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl89k\" (UniqueName: \"kubernetes.io/projected/49fee83f-2189-40b6-8559-022d6cc91770-kube-api-access-vl89k\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.159694 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159673    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-cert\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.159795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.159700    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqw9\" (UniqueName: \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.260560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260526    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-metrics-certs\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.260560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260563    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61eb856b-5461-45bb-9126-c9d33d325ae7-cert\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260590    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htp4f\" (UniqueName: \"kubernetes.io/projected/11772d42-58e7-41ad-8adb-28220a88c2f9-kube-api-access-htp4f\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260635    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl89k\" (UniqueName: \"kubernetes.io/projected/49fee83f-2189-40b6-8559-022d6cc91770-kube-api-access-vl89k\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260677    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-cert\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260703    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqw9\" (UniqueName: \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260768    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcx5\" (UniqueName: \"kubernetes.io/projected/61eb856b-5461-45bb-9126-c9d33d325ae7-kube-api-access-kbcx5\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.260810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.260793    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fee83f-2189-40b6-8559-022d6cc91770-tls-certs\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.263434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.263402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-metrics-certs\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.263580 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.263555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61eb856b-5461-45bb-9126-c9d33d325ae7-cert\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.263800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.263779    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.263840 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.263810    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49fee83f-2189-40b6-8559-022d6cc91770-tls-certs\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.264203 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.264185    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11772d42-58e7-41ad-8adb-28220a88c2f9-cert\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.271911 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.271890    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqw9\" (UniqueName: \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\") pod \"llmisvc-controller-manager-869d5dcc5d-t9tw2\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.276390 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.276362    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl89k\" (UniqueName: \"kubernetes.io/projected/49fee83f-2189-40b6-8559-022d6cc91770-kube-api-access-vl89k\") pod \"model-serving-api-75bd89fdfc-knbqk\" (UID: \"49fee83f-2189-40b6-8559-022d6cc91770\") " pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.279252 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.279230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htp4f\" (UniqueName: \"kubernetes.io/projected/11772d42-58e7-41ad-8adb-28220a88c2f9-kube-api-access-htp4f\") pod \"odh-model-controller-769875db56-ctxbc\" (UID: \"11772d42-58e7-41ad-8adb-28220a88c2f9\") " pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.280360 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.280330    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcx5\" (UniqueName: \"kubernetes.io/projected/61eb856b-5461-45bb-9126-c9d33d325ae7-kube-api-access-kbcx5\") pod \"kserve-controller-manager-776cc66d47-mpgsb\" (UID: \"61eb856b-5461-45bb-9126-c9d33d325ae7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.361651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.361617    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:26.376403 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.376378    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:26.399884 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.399853    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:26.409115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.409081    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:26.657956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.657926    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:46:26.659497 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:26.659459    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5a5cbb08_93fb_453e_bf06_799ee87ed9c6.slice/crio-a5349101b64d381b795eaed9df408266a67a8b4485d60db5ab69266db578df81 WatchSource:0}: Error finding container a5349101b64d381b795eaed9df408266a67a8b4485d60db5ab69266db578df81: Status 404 returned error can't find the container with id a5349101b64d381b795eaed9df408266a67a8b4485d60db5ab69266db578df81
Nov 03 00:46:26.668539 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.668517    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mpgsb"]
Nov 03 00:46:26.669931 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:26.669910    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61eb856b_5461_45bb_9126_c9d33d325ae7.slice/crio-140604577ca3fe92a440d4aa50d9b77b70c4a22be82f04b202a4a69c1c0244d8 WatchSource:0}: Error finding container 140604577ca3fe92a440d4aa50d9b77b70c4a22be82f04b202a4a69c1c0244d8: Status 404 returned error can't find the container with id 140604577ca3fe92a440d4aa50d9b77b70c4a22be82f04b202a4a69c1c0244d8
Nov 03 00:46:26.709232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.709191    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ctxbc"]
Nov 03 00:46:26.709772 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:26.709739    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11772d42_58e7_41ad_8adb_28220a88c2f9.slice/crio-22c9a9e774fc5b6bceb871bf224c6377d109b79384664ababcac1e5a09f0bc0d WatchSource:0}: Error finding container 22c9a9e774fc5b6bceb871bf224c6377d109b79384664ababcac1e5a09f0bc0d: Status 404 returned error can't find the container with id 22c9a9e774fc5b6bceb871bf224c6377d109b79384664ababcac1e5a09f0bc0d
Nov 03 00:46:26.731176 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.731140    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knbqk"]
Nov 03 00:46:26.732422 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:26.732394    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fee83f_2189_40b6_8559_022d6cc91770.slice/crio-07823438f44866a73cbf1cee92405a49c2b801421ef1f0e5dfd8470d4aabbe90 WatchSource:0}: Error finding container 07823438f44866a73cbf1cee92405a49c2b801421ef1f0e5dfd8470d4aabbe90: Status 404 returned error can't find the container with id 07823438f44866a73cbf1cee92405a49c2b801421ef1f0e5dfd8470d4aabbe90
Nov 03 00:46:26.842666 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.842625    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb" event={"ID":"61eb856b-5461-45bb-9126-c9d33d325ae7","Type":"ContainerStarted","Data":"140604577ca3fe92a440d4aa50d9b77b70c4a22be82f04b202a4a69c1c0244d8"}
Nov 03 00:46:26.844114 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.844078    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ctxbc" event={"ID":"11772d42-58e7-41ad-8adb-28220a88c2f9","Type":"ContainerStarted","Data":"22c9a9e774fc5b6bceb871bf224c6377d109b79384664ababcac1e5a09f0bc0d"}
Nov 03 00:46:26.845610 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.845579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" event={"ID":"5a5cbb08-93fb-453e-bf06-799ee87ed9c6","Type":"ContainerStarted","Data":"a5349101b64d381b795eaed9df408266a67a8b4485d60db5ab69266db578df81"}
Nov 03 00:46:26.846985 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:26.846957    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk" event={"ID":"49fee83f-2189-40b6-8559-022d6cc91770","Type":"ContainerStarted","Data":"07823438f44866a73cbf1cee92405a49c2b801421ef1f0e5dfd8470d4aabbe90"}
Nov 03 00:46:27.858045 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:27.858008    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v" event={"ID":"074e788b-3f4c-4b98-a302-c67107a603bc","Type":"ContainerStarted","Data":"946be86abb8a4359d5c24841eb6ad40769f6f1f8025b41ecf832ca5fdd4a6108"}
Nov 03 00:46:27.858892 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:27.858868    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:27.877695 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:27.877640    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v" podStartSLOduration=1.788238406 podStartE2EDuration="8.877622322s" totalImagesPullingTime="7.089383916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:27.87502904 +0000 UTC m=+333.088772715" watchObservedRunningTime="2026-11-03 00:46:27.877622322 +0000 UTC m=+333.091365995"
Nov 03 00:46:29.883140 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:29.883053    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk" event={"ID":"49fee83f-2189-40b6-8559-022d6cc91770","Type":"ContainerStarted","Data":"0ab9621218fc484c4ef5eb7d067f7dd245c6a85bdd1496bfdc2fbf9664c8900f"}
Nov 03 00:46:29.883632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:29.883147    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:29.906200 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:29.906130    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk" podStartSLOduration=1.022586791 podStartE2EDuration="3.906111539s" totalImagesPullingTime="2.883524748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:29.90535669 +0000 UTC m=+335.119100363" watchObservedRunningTime="2026-11-03 00:46:29.906111539 +0000 UTC m=+335.119855214"
Nov 03 00:46:30.900900 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.900864    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb" event={"ID":"61eb856b-5461-45bb-9126-c9d33d325ae7","Type":"ContainerStarted","Data":"b23399c1e3480f8a0fe451531a1b5001afd9a896c92e69ebffd55ba14b97889a"}
Nov 03 00:46:30.901293 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.900917    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:46:30.901293 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.900933    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ctxbc" event={"ID":"11772d42-58e7-41ad-8adb-28220a88c2f9","Type":"ContainerStarted","Data":"13fa64c281963a89a1b5d2558460b7490696cfb3701ec892229f59a5bfb8f174"}
Nov 03 00:46:30.901293 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.900950    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:30.901293 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.900964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" event={"ID":"5a5cbb08-93fb-453e-bf06-799ee87ed9c6","Type":"ContainerStarted","Data":"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"}
Nov 03 00:46:30.912991 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.912921    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb" podStartSLOduration=1.261244528 podStartE2EDuration="4.912909045s" totalImagesPullingTime="3.651664517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:30.912154377 +0000 UTC m=+336.125898050" watchObservedRunningTime="2026-11-03 00:46:30.912909045 +0000 UTC m=+336.126652716"
Nov 03 00:46:30.929432 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.929373    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ctxbc" podStartSLOduration=0.849550129 podStartE2EDuration="4.92935522s" totalImagesPullingTime="4.079805091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:30.92786768 +0000 UTC m=+336.141611356" watchObservedRunningTime="2026-11-03 00:46:30.92935522 +0000 UTC m=+336.143098895"
Nov 03 00:46:30.946202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:30.946136    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" podStartSLOduration=1.4677128179999999 podStartE2EDuration="4.946123335s" totalImagesPullingTime="3.478410517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:30.943803585 +0000 UTC m=+336.157547258" watchObservedRunningTime="2026-11-03 00:46:30.946123335 +0000 UTC m=+336.159867014"
Nov 03 00:46:31.905042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:31.905011    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:46:34.911555 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:34.911527    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kt22v"
Nov 03 00:46:39.877153 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:46:39.877117    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:46:41.906894 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:41.906864    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ctxbc"
Nov 03 00:46:41.909439 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:41.909416    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-knbqk"
Nov 03 00:46:47.280977 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.280502    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"]
Nov 03 00:46:47.304098 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.304062    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"]
Nov 03 00:46:47.304276 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.304212    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.307867 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.307843    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 00:46:47.308825 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.308800    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 00:46:47.308943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.308864    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 00:46:47.308943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.308880    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-x5dcj\""
Nov 03 00:46:47.465064 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.465032    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzbq\" (UniqueName: \"kubernetes.io/projected/2432b11a-588d-42c8-9698-1787ca42ed40-kube-api-access-hbzbq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.465259 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.465092    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.465259 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.465240    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2432b11a-588d-42c8-9698-1787ca42ed40-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.465366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.465274    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.566752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.566666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.566752 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.566735    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzbq\" (UniqueName: \"kubernetes.io/projected/2432b11a-588d-42c8-9698-1787ca42ed40-kube-api-access-hbzbq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.566981 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.566767    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.566981 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.566804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2432b11a-588d-42c8-9698-1787ca42ed40-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.567543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.567510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.567696 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.567681    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2432b11a-588d-42c8-9698-1787ca42ed40-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.569253 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.569226    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2432b11a-588d-42c8-9698-1787ca42ed40-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.574898 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.574872    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzbq\" (UniqueName: \"kubernetes.io/projected/2432b11a-588d-42c8-9698-1787ca42ed40-kube-api-access-hbzbq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zzws2\" (UID: \"2432b11a-588d-42c8-9698-1787ca42ed40\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.619732 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.619692    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:46:47.893938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.893897    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"]
Nov 03 00:46:47.894998 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:46:47.894972    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2432b11a_588d_42c8_9698_1787ca42ed40.slice/crio-6ec8294a3396ce587be109c919da681fca960f906c3a62f8bcbf52b4afa2381c WatchSource:0}: Error finding container 6ec8294a3396ce587be109c919da681fca960f906c3a62f8bcbf52b4afa2381c: Status 404 returned error can't find the container with id 6ec8294a3396ce587be109c919da681fca960f906c3a62f8bcbf52b4afa2381c
Nov 03 00:46:47.993900 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:46:47.993865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2" event={"ID":"2432b11a-588d-42c8-9698-1787ca42ed40","Type":"ContainerStarted","Data":"6ec8294a3396ce587be109c919da681fca960f906c3a62f8bcbf52b4afa2381c"}
Nov 03 00:47:01.910683 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:01.910649    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 00:47:02.911533 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:02.911489    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:47:04.096533 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:04.096499    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2" event={"ID":"2432b11a-588d-42c8-9698-1787ca42ed40","Type":"ContainerStarted","Data":"54e28e3b29e9bbd7515f9bbc1f821c6fba967bc7170356654df6c93322b49c20"}
Nov 03 00:47:04.096932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:04.096571    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:47:04.114642 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:04.114578    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2" podStartSLOduration=1.556941771 podStartE2EDuration="17.114560028s" totalImagesPullingTime="15.557618257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:04.112293094 +0000 UTC m=+369.326036748" watchObservedRunningTime="2026-11-03 00:47:04.114560028 +0000 UTC m=+369.328303701"
Nov 03 00:47:09.878604 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:47:09.878568    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:47:15.107725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:15.107691    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zzws2"
Nov 03 00:47:25.978336 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:25.978299    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:47:25.978777 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:25.978371    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:47:27.238588 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.238496    4531 generic.go:356] "Generic (PLEG): container finished" podID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerID="761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed" exitCode=0
Nov 03 00:47:27.238588 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.238546    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerDied","Data":"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"}
Nov 03 00:47:27.239028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.238934    4531 scope.go:122] "RemoveContainer" containerID="761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"
Nov 03 00:47:27.240247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.240219    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:47:27.240316 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.240290    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:47:27.240353 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.240328    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:47:27.368420 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.368382    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 00:47:27.372273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.372245    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.382104 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.382072    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 00:47:27.424015 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.423976    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 00:47:27.424015 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.423996    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfcxl\""
Nov 03 00:47:27.424015 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.423976    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 00:47:27.441633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.441595    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.441826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.441642    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.441826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.441776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2t68\" (UniqueName: \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.441966 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.441940    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.542650 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.542563    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.542650 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.542607    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.542650 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.542648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2t68\" (UniqueName: \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.542918 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.542732    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.543541 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.543510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.545301 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.545275    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.545479 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.545463    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.553586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.553555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2t68\" (UniqueName: \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\") pod \"maas-api-55cc657fcb-sst8r\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") " pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.689505 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.689464    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:27.988960 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:27.988928    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 00:47:27.990764 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:47:27.990738    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3c4620_dca5_46ac_8e75_8088927931c0.slice/crio-b868999699f37194752f3163e274efe05d309945f65e80be25fa7fc45bbcd0a4 WatchSource:0}: Error finding container b868999699f37194752f3163e274efe05d309945f65e80be25fa7fc45bbcd0a4: Status 404 returned error can't find the container with id b868999699f37194752f3163e274efe05d309945f65e80be25fa7fc45bbcd0a4
Nov 03 00:47:28.255279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:28.255192    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" event={"ID":"4f3c4620-dca5-46ac-8e75-8088927931c0","Type":"ContainerStarted","Data":"b868999699f37194752f3163e274efe05d309945f65e80be25fa7fc45bbcd0a4"}
Nov 03 00:47:28.257520 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:28.257494    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerStarted","Data":"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"}
Nov 03 00:47:28.845690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:28.845591    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:47:28.846340 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:28.845985    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 00:47:28.846340 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:28.846042    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 00:47:29.852367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:29.852300    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 00:47:29.852873 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:29.852399    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 00:47:30.603351 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.601951    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"]
Nov 03 00:47:30.660137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.660101    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"]
Nov 03 00:47:30.660313 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.660235    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.783232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.783199    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d36a864-53d0-4e75-b3bc-867fc1d42a89-cert\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.783232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.783241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbx9\" (UniqueName: \"kubernetes.io/projected/8d36a864-53d0-4e75-b3bc-867fc1d42a89-kube-api-access-lsbx9\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.849186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.848645    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 00:47:30.849186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.848709    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 00:47:30.884641 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.884550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d36a864-53d0-4e75-b3bc-867fc1d42a89-cert\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.884641 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.884606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbx9\" (UniqueName: \"kubernetes.io/projected/8d36a864-53d0-4e75-b3bc-867fc1d42a89-kube-api-access-lsbx9\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.887363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.887335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d36a864-53d0-4e75-b3bc-867fc1d42a89-cert\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:30.893934 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:30.893907    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbx9\" (UniqueName: \"kubernetes.io/projected/8d36a864-53d0-4e75-b3bc-867fc1d42a89-kube-api-access-lsbx9\") pod \"llmisvc-controller-manager-6466c554d8-kkdsj\" (UID: \"8d36a864-53d0-4e75-b3bc-867fc1d42a89\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:31.003671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:31.003637    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:31.354594 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:31.354572    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"]
Nov 03 00:47:31.396091 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:47:31.396050    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d36a864_53d0_4e75_b3bc_867fc1d42a89.slice/crio-5539863dbbb34a59c27766ee8b7373b8153b1252937be08f548192c86c964532 WatchSource:0}: Error finding container 5539863dbbb34a59c27766ee8b7373b8153b1252937be08f548192c86c964532: Status 404 returned error can't find the container with id 5539863dbbb34a59c27766ee8b7373b8153b1252937be08f548192c86c964532
Nov 03 00:47:31.848121 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:31.847632    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 00:47:31.848121 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:31.847704    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 00:47:32.285484 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.285444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj" event={"ID":"8d36a864-53d0-4e75-b3bc-867fc1d42a89","Type":"ContainerStarted","Data":"d76cf028703da0f56edfb71ac08359c730ae59f49a9fa2729ebe9a7b16a68b12"}
Nov 03 00:47:32.285937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.285509    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj" event={"ID":"8d36a864-53d0-4e75-b3bc-867fc1d42a89","Type":"ContainerStarted","Data":"5539863dbbb34a59c27766ee8b7373b8153b1252937be08f548192c86c964532"}
Nov 03 00:47:32.285937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.285629    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:47:32.308991 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.308935    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj" podStartSLOduration=1.5355957789999999 podStartE2EDuration="2.30891833s" totalImagesPullingTime="773.322551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:32.305834238 +0000 UTC m=+397.519577910" watchObservedRunningTime="2026-11-03 00:47:32.30891833 +0000 UTC m=+397.522662070"
Nov 03 00:47:32.844235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.842527    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 00:47:32.844235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:32.842600    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 00:47:33.293690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.293577    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" event={"ID":"4f3c4620-dca5-46ac-8e75-8088927931c0","Type":"ContainerStarted","Data":"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"}
Nov 03 00:47:33.294227 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.293893    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:33.324950 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.324886    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" podStartSLOduration=1.837230049 podStartE2EDuration="6.32486902s" totalImagesPullingTime="4.487638971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:33.322519283 +0000 UTC m=+398.536262955" watchObservedRunningTime="2026-11-03 00:47:33.32486902 +0000 UTC m=+398.538612692"
Nov 03 00:47:33.846425 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.846386    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:47:33.846704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.846681    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:47:33.847594 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:33.847577    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 00:47:39.304345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:47:39.304259    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 00:47:39.878560 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:47:39.878523    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:48:03.296029 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.295997    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-kkdsj"
Nov 03 00:48:03.343998 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.343963    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:48:03.344292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.344262    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" podUID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" containerName="manager" containerID="cri-o://ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f" gracePeriod=30
Nov 03 00:48:03.592225 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.592199    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:48:03.695077 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.695041    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\" (UniqueName: \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\") pod \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") "
Nov 03 00:48:03.695274 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.695259    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\" (UniqueName: \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\") pod \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\" (UID: \"5a5cbb08-93fb-453e-bf06-799ee87ed9c6\") "
Nov 03 00:48:03.697450 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.697420    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert" pod "5a5cbb08-93fb-453e-bf06-799ee87ed9c6" (UID: "5a5cbb08-93fb-453e-bf06-799ee87ed9c6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:48:03.697559 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.697535    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9" pod "5a5cbb08-93fb-453e-bf06-799ee87ed9c6" (UID: "5a5cbb08-93fb-453e-bf06-799ee87ed9c6"). InnerVolumeSpecName "kube-api-access-lgqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:03.796001 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.795963    4531 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:48:03.796001 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:03.796003    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqw9\" (UniqueName: \"kubernetes.io/projected/5a5cbb08-93fb-453e-bf06-799ee87ed9c6-kube-api-access-lgqw9\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:48:04.463179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.463127    4531 generic.go:356] "Generic (PLEG): container finished" podID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" containerID="ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f" exitCode=0
Nov 03 00:48:04.463585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.463203    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"
Nov 03 00:48:04.463585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.463223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" event={"ID":"5a5cbb08-93fb-453e-bf06-799ee87ed9c6","Type":"ContainerDied","Data":"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"}
Nov 03 00:48:04.463585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.463285    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2" event={"ID":"5a5cbb08-93fb-453e-bf06-799ee87ed9c6","Type":"ContainerDied","Data":"a5349101b64d381b795eaed9df408266a67a8b4485d60db5ab69266db578df81"}
Nov 03 00:48:04.463585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.463316    4531 scope.go:122] "RemoveContainer" containerID="ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"
Nov 03 00:48:04.474534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.474509    4531 scope.go:122] "RemoveContainer" containerID="ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"
Nov 03 00:48:04.474867 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:48:04.474834    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f\": container with ID starting with ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f not found: ID does not exist" containerID="ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"
Nov 03 00:48:04.474926 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.474876    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f"} err="failed to get container status \"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f\": rpc error: code = NotFound desc = could not find container \"ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f\": container with ID starting with ef28bf0a5c4f419deca2cf7555a4f478ff6814c5cb9beb6cc4e30a2a5a2b455f not found: ID does not exist"
Nov 03 00:48:04.489614 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.489587    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:48:04.494025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.494001    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t9tw2"]
Nov 03 00:48:04.902754 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:48:04.902723    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" path="/var/lib/kubelet/pods/5a5cbb08-93fb-453e-bf06-799ee87ed9c6/volumes"
Nov 03 00:48:09.876132 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:48:09.876086    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:48:39.881388 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:48:39.881341    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:49:09.876331 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:49:09.876234    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:49:13.978461 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.978426    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"]
Nov 03 00:49:13.978888 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.978873    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" containerName="manager"
Nov 03 00:49:13.978931 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.978889    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" containerName="manager"
Nov 03 00:49:13.978972 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.978962    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5cbb08-93fb-453e-bf06-799ee87ed9c6" containerName="manager"
Nov 03 00:49:13.982322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.982298    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:13.999090 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:13.999064    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"]
Nov 03 00:49:14.062114 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062077    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062140    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9342480-831c-4417-8711-a69e79b817a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062305    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062349    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062392    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062578 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9342480-831c-4417-8711-a69e79b817a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062578 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzczb\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-kube-api-access-pzczb\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.062673 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.062592    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.163743 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163706    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.163743 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9342480-831c-4417-8711-a69e79b817a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163773    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzczb\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-kube-api-access-pzczb\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163791    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163830    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9342480-831c-4417-8711-a69e79b817a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163923    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163955    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.163984    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.164217    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164520 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.164498    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164520 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.164506    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.164548    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.164742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.164719    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9342480-831c-4417-8711-a69e79b817a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.166233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.166204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9342480-831c-4417-8711-a69e79b817a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.166516 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.166496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9342480-831c-4417-8711-a69e79b817a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.172727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.172697    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.172871 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.172849    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzczb\" (UniqueName: \"kubernetes.io/projected/c9342480-831c-4417-8711-a69e79b817a6-kube-api-access-pzczb\") pod \"maas-default-gateway-openshift-default-596dc9545f-d852p\" (UID: \"c9342480-831c-4417-8711-a69e79b817a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.297471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.297382    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:14.579251 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.579218    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"]
Nov 03 00:49:14.580900 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:49:14.580869    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9342480_831c_4417_8711_a69e79b817a6.slice/crio-15ce91ff33140bff1b1f1f193f9f7452e17d5fc199c51b6922203bfdb4c7120d WatchSource:0}: Error finding container 15ce91ff33140bff1b1f1f193f9f7452e17d5fc199c51b6922203bfdb4c7120d: Status 404 returned error can't find the container with id 15ce91ff33140bff1b1f1f193f9f7452e17d5fc199c51b6922203bfdb4c7120d
Nov 03 00:49:14.583145 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.583100    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:14.583288 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.583229    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:14.583335 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.583298    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:14.865445 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.865360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" event={"ID":"c9342480-831c-4417-8711-a69e79b817a6","Type":"ContainerStarted","Data":"78a0364dd942eacc8e2d89d5633ee0143c153881667053a92472c1a82a57d024"}
Nov 03 00:49:14.865599 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.865460    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" event={"ID":"c9342480-831c-4417-8711-a69e79b817a6","Type":"ContainerStarted","Data":"15ce91ff33140bff1b1f1f193f9f7452e17d5fc199c51b6922203bfdb4c7120d"}
Nov 03 00:49:14.914660 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:14.914608    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podStartSLOduration=1.914593028 podStartE2EDuration="1.914593028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:14.910429053 +0000 UTC m=+500.124172727" watchObservedRunningTime="2026-11-03 00:49:14.914593028 +0000 UTC m=+500.128336786"
Nov 03 00:49:15.298214 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:15.298103    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:15.299632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:15.299606    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:15.299742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:15.299677    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:16.297939 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:16.297891    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:16.298121 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:16.297960    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:17.298210 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:17.298144    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:17.298610 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:17.298236    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:18.298323 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:18.298286    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:18.298756 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:18.298375    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:19.297990 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:19.297957    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:19.298275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:19.298015    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:20.298463 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:20.298428    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:20.298844 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:20.298481    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:21.297801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:21.297759    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:49:21.297994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:21.297816    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 00:49:22.754076 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:22.754038    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:22.754586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:22.754485    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:22.755347 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:22.755322    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p"
Nov 03 00:49:39.876073 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:49:39.876033    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:49:57.833589 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.833552    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:49:57.837578 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.837540    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:57.842138 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.842104    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15abbf02-serving-cert\""
Nov 03 00:49:57.847607 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.847121    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:49:57.914766 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.914732    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcq86\" (UniqueName: \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:57.914940 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.914774    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:57.914989 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.914932    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:57.915041 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:57.914996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.016084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.016039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.016323 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.016099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.016323 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.016192    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcq86\" (UniqueName: \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.016323 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:49:58.016212    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-serving-cert: secret "maas-api-e2e-ait-15abbf02-serving-cert" not found
Nov 03 00:49:58.016323 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.016241    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.016323 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:49:58.016276    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls podName:1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0 nodeName:}" failed. No retries permitted until 2026-11-03 00:49:58.516259913 +0000 UTC m=+543.730003582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls") pod "maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" (UID: "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0") : secret "maas-api-e2e-ait-15abbf02-serving-cert" not found
Nov 03 00:49:58.016934 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.016916    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.018839 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.018807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.030379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.030346    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcq86\" (UniqueName: \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.521153 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.521100    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.523716 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.523688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") pod \"maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:58.763078 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:58.763042    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:49:59.059874 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:59.059842    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:49:59.061657 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:49:59.061630    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3a3cc3_80a5_42ae_bd39_5f23323d2cf0.slice/crio-be2e5495887cbaf8ba07fd0e38d4dcb8c0e7ebb872faea04dd91f25e7ced9ea2 WatchSource:0}: Error finding container be2e5495887cbaf8ba07fd0e38d4dcb8c0e7ebb872faea04dd91f25e7ced9ea2: Status 404 returned error can't find the container with id be2e5495887cbaf8ba07fd0e38d4dcb8c0e7ebb872faea04dd91f25e7ced9ea2
Nov 03 00:49:59.108669 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:49:59.108624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" event={"ID":"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0","Type":"ContainerStarted","Data":"be2e5495887cbaf8ba07fd0e38d4dcb8c0e7ebb872faea04dd91f25e7ced9ea2"}
Nov 03 00:50:00.115822 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:00.115776    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" event={"ID":"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0","Type":"ContainerStarted","Data":"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"}
Nov 03 00:50:00.116245 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:00.115852    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:50:00.140365 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:00.140294    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" podStartSLOduration=2.70001311 podStartE2EDuration="3.140273866s" totalImagesPullingTime="440.260756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:00.136198933 +0000 UTC m=+545.349942606" watchObservedRunningTime="2026-11-03 00:50:00.140273866 +0000 UTC m=+545.354017540"
Nov 03 00:50:01.219533 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.219497    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:01.224036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.224009    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.232840 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.232797    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c83251ee-serving-cert\""
Nov 03 00:50:01.244494 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.244463    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:01.352885 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.352839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.353097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.352909    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.353097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.353032    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.353263 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.353114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hx4\" (UniqueName: \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.454860 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.454817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.455054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.454894    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.455054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.454921    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.455054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.454940    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hx4\" (UniqueName: \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.455860 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.455828    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.457811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.457787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.457912 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.457787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.470496 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.470435    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hx4\" (UniqueName: \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\") pod \"maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.542875 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.542833    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:01.866892 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:01.866863    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:01.867961 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:50:01.867936    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fa5bc9_e1b7_4285_8c49_921734dc7872.slice/crio-d12e2c621d00245770a428d958358e8003287bd185cbe023f0ca254e4cc68ee7 WatchSource:0}: Error finding container d12e2c621d00245770a428d958358e8003287bd185cbe023f0ca254e4cc68ee7: Status 404 returned error can't find the container with id d12e2c621d00245770a428d958358e8003287bd185cbe023f0ca254e4cc68ee7
Nov 03 00:50:02.128306 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:02.128204    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" event={"ID":"56fa5bc9-e1b7-4285-8c49-921734dc7872","Type":"ContainerStarted","Data":"d12e2c621d00245770a428d958358e8003287bd185cbe023f0ca254e4cc68ee7"}
Nov 03 00:50:03.136331 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:03.136288    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" event={"ID":"56fa5bc9-e1b7-4285-8c49-921734dc7872","Type":"ContainerStarted","Data":"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"}
Nov 03 00:50:03.136848 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:03.136429    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:03.157690 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:03.157637    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" podStartSLOduration=1.648019911 podStartE2EDuration="2.157623263s" totalImagesPullingTime="509.603352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:03.15631834 +0000 UTC m=+548.370062006" watchObservedRunningTime="2026-11-03 00:50:03.157623263 +0000 UTC m=+548.371366935"
Nov 03 00:50:06.127840 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:06.127810    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:50:09.148283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:09.148251    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:09.878575 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:09.878536    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:50:28.385093 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.384479    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:50:28.385947 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.385915    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" podUID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" containerName="maas-api" containerID="cri-o://15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246" gracePeriod=30
Nov 03 00:50:28.699184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.675344    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:50:28.759814 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.759780    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\") pod \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") "
Nov 03 00:50:28.759982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.759888    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\" (UniqueName: \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\") pod \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") "
Nov 03 00:50:28.759982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.759923    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") pod \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") "
Nov 03 00:50:28.760054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.759977    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\") pod \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\" (UID: \"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0\") "
Nov 03 00:50:28.760467 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.760434    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca" pod "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" (UID: "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:28.762107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.762074    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs" pod "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" (UID: "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:28.763102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.762615    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls" pod "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" (UID: "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:28.763102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.762663    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86" pod "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" (UID: "1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0"). InnerVolumeSpecName "kube-api-access-fcq86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:28.860937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.860897    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:28.860937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.860934    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcq86\" (UniqueName: \"kubernetes.io/projected/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-kube-api-access-fcq86\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:28.861196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.860949    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:28.861196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:28.860963    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:29.291038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.290949    4531 generic.go:356] "Generic (PLEG): container finished" podID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" containerID="15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246" exitCode=0
Nov 03 00:50:29.291038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.291012    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"
Nov 03 00:50:29.291379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.291028    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" event={"ID":"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0","Type":"ContainerDied","Data":"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"}
Nov 03 00:50:29.291379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.291099    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z" event={"ID":"1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0","Type":"ContainerDied","Data":"be2e5495887cbaf8ba07fd0e38d4dcb8c0e7ebb872faea04dd91f25e7ced9ea2"}
Nov 03 00:50:29.291379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.291129    4531 scope.go:122] "RemoveContainer" containerID="15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"
Nov 03 00:50:29.301590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.301568    4531 scope.go:122] "RemoveContainer" containerID="15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"
Nov 03 00:50:29.302374 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:29.302349    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246\": container with ID starting with 15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246 not found: ID does not exist" containerID="15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"
Nov 03 00:50:29.302495 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.302381    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246"} err="failed to get container status \"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246\": rpc error: code = NotFound desc = could not find container \"15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246\": container with ID starting with 15d0381d6751c3f39cbc272bce75909dcde0b8905a0423129f3a7a7ea29a5246 not found: ID does not exist"
Nov 03 00:50:29.311382 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.311350    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:50:29.313570 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.313547    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15abbf02-66b9ccbdf8-qk27z"]
Nov 03 00:50:29.698483 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.698449    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:29.698938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.698715    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" podUID="56fa5bc9-e1b7-4285-8c49-921734dc7872" containerName="maas-api" containerID="cri-o://7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf" gracePeriod=30
Nov 03 00:50:29.965729 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:29.965650    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:30.075354 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075316    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\") pod \"56fa5bc9-e1b7-4285-8c49-921734dc7872\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") "
Nov 03 00:50:30.075558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075369    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\" (UniqueName: \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\") pod \"56fa5bc9-e1b7-4285-8c49-921734dc7872\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") "
Nov 03 00:50:30.075558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075476    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\") pod \"56fa5bc9-e1b7-4285-8c49-921734dc7872\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") "
Nov 03 00:50:30.075558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075521    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\") pod \"56fa5bc9-e1b7-4285-8c49-921734dc7872\" (UID: \"56fa5bc9-e1b7-4285-8c49-921734dc7872\") "
Nov 03 00:50:30.075766 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075737    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca" pod "56fa5bc9-e1b7-4285-8c49-921734dc7872" (UID: "56fa5bc9-e1b7-4285-8c49-921734dc7872"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:30.075906 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.075887    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5bc9-e1b7-4285-8c49-921734dc7872-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:30.077795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.077688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls" pod "56fa5bc9-e1b7-4285-8c49-921734dc7872" (UID: "56fa5bc9-e1b7-4285-8c49-921734dc7872"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:30.077795 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.077757    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs" pod "56fa5bc9-e1b7-4285-8c49-921734dc7872" (UID: "56fa5bc9-e1b7-4285-8c49-921734dc7872"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:30.078090 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.078061    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4" pod "56fa5bc9-e1b7-4285-8c49-921734dc7872" (UID: "56fa5bc9-e1b7-4285-8c49-921734dc7872"). InnerVolumeSpecName "kube-api-access-k6hx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:30.177190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.177146    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:30.177190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.177189    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5bc9-e1b7-4285-8c49-921734dc7872-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:30.177472 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.177200    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hx4\" (UniqueName: \"kubernetes.io/projected/56fa5bc9-e1b7-4285-8c49-921734dc7872-kube-api-access-k6hx4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:50:30.299052 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.298955    4531 generic.go:356] "Generic (PLEG): container finished" podID="56fa5bc9-e1b7-4285-8c49-921734dc7872" containerID="7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf" exitCode=0
Nov 03 00:50:30.299052 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.299007    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"
Nov 03 00:50:30.299052 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.299011    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" event={"ID":"56fa5bc9-e1b7-4285-8c49-921734dc7872","Type":"ContainerDied","Data":"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"}
Nov 03 00:50:30.299367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.299080    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm" event={"ID":"56fa5bc9-e1b7-4285-8c49-921734dc7872","Type":"ContainerDied","Data":"d12e2c621d00245770a428d958358e8003287bd185cbe023f0ca254e4cc68ee7"}
Nov 03 00:50:30.299367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.299110    4531 scope.go:122] "RemoveContainer" containerID="7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"
Nov 03 00:50:30.314240 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.314103    4531 scope.go:122] "RemoveContainer" containerID="7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"
Nov 03 00:50:30.315442 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:30.314594    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf\": container with ID starting with 7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf not found: ID does not exist" containerID="7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"
Nov 03 00:50:30.315442 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.314630    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf"} err="failed to get container status \"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf\": rpc error: code = NotFound desc = could not find container \"7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf\": container with ID starting with 7884cd698e906541d115bde9b63d79ad1466f67476bb63249be35e6d44853cbf not found: ID does not exist"
Nov 03 00:50:30.329212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.326493    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:30.337001 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.336969    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c83251ee-65c74ffb79-nwtlm"]
Nov 03 00:50:30.903079 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.903044    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" path="/var/lib/kubelet/pods/1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0/volumes"
Nov 03 00:50:30.903629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:30.903602    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fa5bc9-e1b7-4285-8c49-921734dc7872" path="/var/lib/kubelet/pods/56fa5bc9-e1b7-4285-8c49-921734dc7872/volumes"
Nov 03 00:50:34.125955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.125917    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126679    4531 cpu_manager.go:414] "containerMap: removing container" podUID="56fa5bc9-e1b7-4285-8c49-921734dc7872" containerName="maas-api"
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126700    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fa5bc9-e1b7-4285-8c49-921734dc7872" containerName="maas-api"
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126716    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" containerName="maas-api"
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126723    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" containerName="maas-api"
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126829    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fa5bc9-e1b7-4285-8c49-921734dc7872" containerName="maas-api"
Nov 03 00:50:34.127643 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.126841    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3a3cc3-80a5-42ae-bd39-5f23323d2cf0" containerName="maas-api"
Nov 03 00:50:34.132328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.132299    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.146241 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.145999    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-650be0-serving-cert\""
Nov 03 00:50:34.156550 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.156481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 00:50:34.224305 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.224260    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnwd\" (UniqueName: \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.224498 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.224318    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.224498 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.224453    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.224580 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.224526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.334974 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.334932    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.335273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.335054    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnwd\" (UniqueName: \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.335273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.335086    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.335273 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.335142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.338684 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.335917    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.344245 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:34.336423    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-serving-cert: secret "maas-api-e2e-worker-w4-650be0-serving-cert" not found
Nov 03 00:50:34.344397 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:34.344323    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls podName:c28a6fd1-b43f-45bd-980c-672dc3d6cc1d nodeName:}" failed. No retries permitted until 2026-11-03 00:50:34.844298364 +0000 UTC m=+580.058042027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls") pod "maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" (UID: "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d") : secret "maas-api-e2e-worker-w4-650be0-serving-cert" not found
Nov 03 00:50:34.356128 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.356098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.358832 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.358802    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnwd\" (UniqueName: \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.942406 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.942360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:34.945269 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:34.945240    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:35.057929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.057341    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:35.234925 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.234844    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 00:50:35.238973 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.238950    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.248737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.248710    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f7680d-serving-cert\""
Nov 03 00:50:35.253409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.253378    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 00:50:35.346262 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.346232    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 00:50:35.348494 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:50:35.348463    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28a6fd1_b43f_45bd_980c_672dc3d6cc1d.slice/crio-4c39c6194437e0d2239807c07ce8988d8b28faf2a6832718a9344fbdfb10a8c2 WatchSource:0}: Error finding container 4c39c6194437e0d2239807c07ce8988d8b28faf2a6832718a9344fbdfb10a8c2: Status 404 returned error can't find the container with id 4c39c6194437e0d2239807c07ce8988d8b28faf2a6832718a9344fbdfb10a8c2
Nov 03 00:50:35.354483 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.354453    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrwq\" (UniqueName: \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.354632 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.354512    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.354781 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.354752    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.354839 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.354803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.360679 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.360644    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" event={"ID":"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d","Type":"ContainerStarted","Data":"4c39c6194437e0d2239807c07ce8988d8b28faf2a6832718a9344fbdfb10a8c2"}
Nov 03 00:50:35.456150 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.456109    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrwq\" (UniqueName: \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.456405 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.456197    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.456405 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.456316    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.456405 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.456349    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.456843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.456822    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.458969 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.458927    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.459102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.458933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.464252 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.464220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrwq\" (UniqueName: \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\") pod \"maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.562345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.562255    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:35.841921 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:35.841887    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 00:50:36.372046 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.372011    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" event={"ID":"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d","Type":"ContainerStarted","Data":"bdb72984720cb0161b922418099093b426ab09d2f2a3ec35be2b2a657bb74701"}
Nov 03 00:50:36.372966 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.372945    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:36.375083 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.375001    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" event={"ID":"a86ecb79-989d-47cb-8661-998ad177164e","Type":"ContainerStarted","Data":"5254a3e9f4c8095582443e1331e06eab9bbb571e9cab126900f3ee45066ac9cb"}
Nov 03 00:50:36.375083 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.375047    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" event={"ID":"a86ecb79-989d-47cb-8661-998ad177164e","Type":"ContainerStarted","Data":"b85dbb9d409ff2c4fcb03a97062202534444dda5e03c0bfc82c377e94eacaa89"}
Nov 03 00:50:36.375288 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.375202    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:36.394412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.394353    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" podStartSLOduration=1.959990412 podStartE2EDuration="2.39433859s" totalImagesPullingTime="434.348178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:36.393093026 +0000 UTC m=+581.606836699" watchObservedRunningTime="2026-11-03 00:50:36.39433859 +0000 UTC m=+581.608082262"
Nov 03 00:50:36.414411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:36.414340    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" podStartSLOduration=0.985378811 podStartE2EDuration="1.414320829s" totalImagesPullingTime="428.942018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:36.412397366 +0000 UTC m=+581.626141039" watchObservedRunningTime="2026-11-03 00:50:36.414320829 +0000 UTC m=+581.628064503"
Nov 03 00:50:39.877672 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:50:39.877591    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:50:42.386889 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:42.386861    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 00:50:43.396981 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:43.396951    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 00:50:46.770790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.770758    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 00:50:46.774503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.774477    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.781324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.781271    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-88bbf7-serving-cert\""
Nov 03 00:50:46.785116 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.785089    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 00:50:46.884755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.884719    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.884938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.884766    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpsn\" (UniqueName: \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.884938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.884864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.884938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.884905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.986012 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.985974    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.986239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.986033    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.986239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.986187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.986239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.986232    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpsn\" (UniqueName: \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.987067 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.987035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.989413 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.989383    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.989583 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.989500    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:46.997089 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:46.997055    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpsn\" (UniqueName: \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\") pod \"maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:47.091101 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:47.091068    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:47.369737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:47.369710    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 00:50:47.373214 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:50:47.373149    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a74844_b9b3_4916_9bbb_f39cfc670094.slice/crio-b7272737b7f29019a69a898b3511e5c9c55aacc0da72d0b577e4d63a9991b1d3 WatchSource:0}: Error finding container b7272737b7f29019a69a898b3511e5c9c55aacc0da72d0b577e4d63a9991b1d3: Status 404 returned error can't find the container with id b7272737b7f29019a69a898b3511e5c9c55aacc0da72d0b577e4d63a9991b1d3
Nov 03 00:50:47.449315 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:47.449264    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" event={"ID":"06a74844-b9b3-4916-9bbb-f39cfc670094","Type":"ContainerStarted","Data":"b7272737b7f29019a69a898b3511e5c9c55aacc0da72d0b577e4d63a9991b1d3"}
Nov 03 00:50:48.457383 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:48.457343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" event={"ID":"06a74844-b9b3-4916-9bbb-f39cfc670094","Type":"ContainerStarted","Data":"db580f8424ed51a4538da55f04a1d1dcb6e2a288a713d1c4d37ccf6b5fb47f04"}
Nov 03 00:50:48.457783 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:48.457408    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:50:48.476063 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:48.476002    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" podStartSLOduration=2.097172279 podStartE2EDuration="2.475983127s" totalImagesPullingTime="378.810848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:48.475780529 +0000 UTC m=+593.689524202" watchObservedRunningTime="2026-11-03 00:50:48.475983127 +0000 UTC m=+593.689726799"
Nov 03 00:50:54.471391 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:50:54.471360    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 00:51:09.880106 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:51:09.880050    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:51:25.947618 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.947578    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 00:51:25.952961 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.952928    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.959780 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.959750    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"kube-root-ca.crt\""
Nov 03 00:51:25.960439 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.960385    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 00:51:25.960439 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.960390    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"premium-w0-2f9a3a-kserve-self-signed-certs\""
Nov 03 00:51:25.960623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.960450    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"openshift-service-ca.crt\""
Nov 03 00:51:25.960623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.960551    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"default-dockercfg-tf6nn\""
Nov 03 00:51:25.996352 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996319    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.996547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.996547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996500    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhr2\" (UniqueName: \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.996672 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996599    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.996727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996682    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:25.996857 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:25.996832    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097437 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097396    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097437 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097444    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097470    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhr2\" (UniqueName: \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097537    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.097971 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097946    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.098028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.097958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.098233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.098214    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.100236 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.100212    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.105827 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.105803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhr2\" (UniqueName: \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.108746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.108721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\") pod \"premium-w0-2f9a3a-kserve-68bd866d8-np5kr\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") " pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.280619 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.280534    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:26.565030 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.565002    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 00:51:26.567659 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:51:26.567613    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9ba8ade_6968_438c_b4f0_3ef913b858ad.slice/crio-fff0723bcb88e84cf470cfe4be0e66b52076c400b968146f369ca1f032bfb4e6 WatchSource:0}: Error finding container fff0723bcb88e84cf470cfe4be0e66b52076c400b968146f369ca1f032bfb4e6: Status 404 returned error can't find the container with id fff0723bcb88e84cf470cfe4be0e66b52076c400b968146f369ca1f032bfb4e6
Nov 03 00:51:26.569772 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.569753    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:51:26.702115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:26.702080    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" event={"ID":"b9ba8ade-6968-438c-b4f0-3ef913b858ad","Type":"ContainerStarted","Data":"fff0723bcb88e84cf470cfe4be0e66b52076c400b968146f369ca1f032bfb4e6"}
Nov 03 00:51:28.719866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:28.719824    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" event={"ID":"b9ba8ade-6968-438c-b4f0-3ef913b858ad","Type":"ContainerStarted","Data":"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"}
Nov 03 00:51:28.738468 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:28.738411    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podStartSLOduration=2.261333514 podStartE2EDuration="3.738395815s" totalImagesPullingTime="1.477062301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:28.737330284 +0000 UTC m=+633.951073963" watchObservedRunningTime="2026-11-03 00:51:28.738395815 +0000 UTC m=+633.952139515"
Nov 03 00:51:29.281424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:29.281386    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:34.107844 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.107809    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 00:51:34.111827 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.111806    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.118640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.118617    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"openshift-service-ca.crt\""
Nov 03 00:51:34.118891 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.118851    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"kube-root-ca.crt\""
Nov 03 00:51:34.119629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.119609    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"default-dockercfg-j2wgx\""
Nov 03 00:51:34.134232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.134202    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"premium-w1-88bbf7-kserve-self-signed-certs\""
Nov 03 00:51:34.141843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.141817    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 00:51:34.185568 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.185531    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.185728 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.185583    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ljn\" (UniqueName: \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.185728 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.185689    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.185837 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.185738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.185957 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.185938    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.186203 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.186003    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287128    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287267    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287298    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ljn\" (UniqueName: \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287379 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.287737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287711    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.288010 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.288010 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.287998    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.288180 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.288066    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.290020 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.289993    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.290201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.290153    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.298648 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.298618    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ljn\" (UniqueName: \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\") pod \"premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") " pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.429031 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.428936    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:34.711226 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.711195    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 00:51:34.712349 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:51:34.712312    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69203221_8dae_4a89_b636_321a7a2442f8.slice/crio-ca2b87a94bc3e37e637488c16f096d1b78b345c9978d0e79161b08fb5267f659 WatchSource:0}: Error finding container ca2b87a94bc3e37e637488c16f096d1b78b345c9978d0e79161b08fb5267f659: Status 404 returned error can't find the container with id ca2b87a94bc3e37e637488c16f096d1b78b345c9978d0e79161b08fb5267f659
Nov 03 00:51:34.762315 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:34.762284    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" event={"ID":"69203221-8dae-4a89-b636-321a7a2442f8","Type":"ContainerStarted","Data":"ca2b87a94bc3e37e637488c16f096d1b78b345c9978d0e79161b08fb5267f659"}
Nov 03 00:51:35.771377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:35.771340    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" event={"ID":"69203221-8dae-4a89-b636-321a7a2442f8","Type":"ContainerStarted","Data":"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"}
Nov 03 00:51:35.790567 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:35.789189    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podStartSLOduration=1.789152429 podStartE2EDuration="1.789152429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:35.788379157 +0000 UTC m=+641.002122829" watchObservedRunningTime="2026-11-03 00:51:35.789152429 +0000 UTC m=+641.002896105"
Nov 03 00:51:36.280956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:36.280919    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:36.294096 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:36.294069    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:36.429112 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:36.429039    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:36.790207 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:36.790177    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:51:39.878814 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:51:39.878779    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:51:44.430311 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:44.430275    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:44.448737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:44.448705    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:51:44.841790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:51:44.841758    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:52:09.876969 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:52:09.876880    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:52:15.653938 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.653899    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 00:52:15.658615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.658589    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.663253 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.663228    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-83755a-openshift-default-dockercfg-qhtrs\""
Nov 03 00:52:15.668585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.668241    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 00:52:15.807715 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.807671    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.807921 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.807728    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.807921 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.807858    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808014 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.807923    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808050 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.808024    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.808090    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfl5c\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808146 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.808123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808288 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.808268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.808389 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.808371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909005 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.908913    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909005 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.908977    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909019    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfl5c\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909174    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909215    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909244    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909539 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909751    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909998 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909824    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.909998 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909847    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.910108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.909995    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.910108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.910013    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.912305 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.912278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.912479 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.912463    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.916955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.916929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.917190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.917145    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfl5c\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:15.977948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:15.977915    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:16.266480 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:16.266447    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 00:52:16.268368 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:52:16.268340    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844f57de_e831_44bf_b3a5_f9532133f011.slice/crio-4e32a3d6bef44577489270bb47221699947bb2b5d6a94221c1694bb082c310b9 WatchSource:0}: Error finding container 4e32a3d6bef44577489270bb47221699947bb2b5d6a94221c1694bb082c310b9: Status 404 returned error can't find the container with id 4e32a3d6bef44577489270bb47221699947bb2b5d6a94221c1694bb082c310b9
Nov 03 00:52:16.271028 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:16.270983    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:16.271100 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:16.271079    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:16.271204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:16.271150    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:17.020894 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.020854    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" event={"ID":"844f57de-e831-44bf-b3a5-f9532133f011","Type":"ContainerStarted","Data":"8f58813b0dd7b8e6fce14a3452b832be7b529a0d140683348eb62b32165e11f0"}
Nov 03 00:52:17.021393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.020910    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" event={"ID":"844f57de-e831-44bf-b3a5-f9532133f011","Type":"ContainerStarted","Data":"4e32a3d6bef44577489270bb47221699947bb2b5d6a94221c1694bb082c310b9"}
Nov 03 00:52:17.056769 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.053548    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podStartSLOduration=2.053526808 podStartE2EDuration="2.053526808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:17.050358106 +0000 UTC m=+682.264101780" watchObservedRunningTime="2026-11-03 00:52:17.053526808 +0000 UTC m=+682.267270480"
Nov 03 00:52:17.978465 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.978428    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:17.979900 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.979871    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:52:17.980051 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:17.979926    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:52:18.978684 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:18.978645    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:52:18.979153 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:18.978701    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:52:19.978996 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:19.978956    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:52:19.979424 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:19.979026    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:52:20.978797 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:20.978759    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:52:20.979000 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:20.978811    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:52:21.979694 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:21.979651    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:52:21.980198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:21.979719    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:52:23.133868 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:23.133832    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:23.134446 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:23.134279    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:23.135135 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:23.135119    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 00:52:26.784258 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.784220    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 00:52:26.788321 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.788301    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.798869 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.798844    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-650be0-openshift-default-dockercfg-l27n6\""
Nov 03 00:52:26.803648 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.803624    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 00:52:26.860239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860198    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860313    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860390    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860537 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860554    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860616 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860659 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860620    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.860702 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.860663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w669\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.961880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.961759    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.961880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.961798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.961952    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.961987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w669\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.962051    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.962119    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.962146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.962238    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.962362    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.963070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.963235    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.963328    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.963391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.963420    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.965302    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.965605    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.970096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:26.974852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:26.970541    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w669\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:27.103574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:27.103529    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:27.388207 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:27.388179    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 00:52:27.389350 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:52:27.389316    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e582b53_d78a_49d2_b5cd_9ecb56d7f331.slice/crio-24dccc77cc644fac11388bdcd916bde8f21115cd0c234f94d7d1bcafac72c754 WatchSource:0}: Error finding container 24dccc77cc644fac11388bdcd916bde8f21115cd0c234f94d7d1bcafac72c754: Status 404 returned error can't find the container with id 24dccc77cc644fac11388bdcd916bde8f21115cd0c234f94d7d1bcafac72c754
Nov 03 00:52:27.392525 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:27.392487    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:27.392726 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:27.392697    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:27.392824 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:27.392755    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:28.138880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:28.138836    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" event={"ID":"0e582b53-d78a-49d2-b5cd-9ecb56d7f331","Type":"ContainerStarted","Data":"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"}
Nov 03 00:52:28.139771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:28.139743    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" event={"ID":"0e582b53-d78a-49d2-b5cd-9ecb56d7f331","Type":"ContainerStarted","Data":"24dccc77cc644fac11388bdcd916bde8f21115cd0c234f94d7d1bcafac72c754"}
Nov 03 00:52:29.106182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.105531    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:29.110420 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.110386    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:29.110586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.110445    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:29.249189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.246868    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podStartSLOduration=3.246846201 podStartE2EDuration="3.246846201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:28.17377949 +0000 UTC m=+693.387523164" watchObservedRunningTime="2026-11-03 00:52:29.246846201 +0000 UTC m=+694.460589875"
Nov 03 00:52:29.321199 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.321142    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 00:52:29.362862 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.360475    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501352    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbft\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501647    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501716    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501757    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.502202 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.501809    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.505282 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.505258    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.604787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.604869    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.604937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.604979    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbft\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605042    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605085    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605190    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.605738    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.606415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.606015    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.612229 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.607803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.614059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.613514    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.614059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.613543    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.644187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.631428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.644187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.632006    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.644187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.634583    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.669003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.668968    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbft\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\") pod \"maas-default-gateway-openshift-default-596dc9545f-9xvbs\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:29.692185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:29.689480    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:30.114289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.114234    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:30.142493 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.134723    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:30.535454 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.533625    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 00:52:30.547269 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.547235    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:30.547454 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.547436    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:30.547570 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:30.547556    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:31.108181 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.107695    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:31.108181 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.107774    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:31.379242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.378676    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" event={"ID":"0a553979-f84e-47b0-8f26-2b70ecbb8cbf","Type":"ContainerStarted","Data":"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"}
Nov 03 00:52:31.379242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.378743    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" event={"ID":"0a553979-f84e-47b0-8f26-2b70ecbb8cbf","Type":"ContainerStarted","Data":"b5e7b6cef90463f636400a291e5e6f298bf3307a625ec0599ea55127b004e738"}
Nov 03 00:52:31.411910 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.411859    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podStartSLOduration=2.411843996 podStartE2EDuration="2.411843996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:31.40792931 +0000 UTC m=+696.621672982" watchObservedRunningTime="2026-11-03 00:52:31.411843996 +0000 UTC m=+696.625587667"
Nov 03 00:52:31.692110 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.692029    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:31.693706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.693676    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:31.693808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:31.693746    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:32.104033 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:32.103990    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:32.104241 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:32.104053    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:32.692572 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:32.692536    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:32.693124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:32.693092    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:33.105198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:33.104710    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:33.105198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:33.104786    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:33.692563 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:33.692530    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:33.692836 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:33.692809    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:34.104743 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:34.104706    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 00:52:34.104915 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:34.104764    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 00:52:34.692264 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:34.692205    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:34.692451 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:34.692288    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:35.108543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:35.108510    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:35.108957 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:35.108858    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:35.109629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:35.109612    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 00:52:35.691993 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:35.691953    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:35.692190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:35.692007    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:36.692311 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:36.692266    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:36.692797 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:36.692348    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:37.691922 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:37.691889    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:37.692134 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:37.691954    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:38.693195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:38.692548    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:52:38.693195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:38.692622    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:52:39.933531 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:52:39.933436    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:52:40.696190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:40.696123    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:52:40.696528 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:40.696494    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:41.738377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:41.738336    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:52:41.738982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:41.738421    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:42.783328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:42.783288    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-9xvbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:52:42.783977 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:42.783944    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:43.076124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:43.075114    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:43.076124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:43.075950    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:43.076860 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:43.076837    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 00:52:51.022107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.022070    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:52:51.026189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.026151    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.032855 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.032836    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"default-dockercfg-ggdgp\""
Nov 03 00:52:51.040948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.036864    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"e2e-conflict-b-45480c9d-kserve-self-signed-certs\""
Nov 03 00:52:51.040948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.036967    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"openshift-service-ca.crt\""
Nov 03 00:52:51.040948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.037026    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"kube-root-ca.crt\""
Nov 03 00:52:51.040948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.040152    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:52:51.078927 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.078890    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.079115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.079020    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.079115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.079070    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4zj\" (UniqueName: \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.079266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.079150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.079266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.079193    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.079266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.079224    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180279    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4zj\" (UniqueName: \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180396    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180426    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180468    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180512    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.180629    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.181059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.181648    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.182292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.181871    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.184721 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.184673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.185888 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.185860    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.188805 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.188784    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4zj\" (UniqueName: \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\") pod \"e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") " pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.347363 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.347015    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:51.866735 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:51.866686    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:52:52.576667 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:52.576628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" event={"ID":"330f4583-b2cd-4953-b5a0-642073169731","Type":"ContainerStarted","Data":"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"}
Nov 03 00:52:52.577342 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:52.577319    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" event={"ID":"330f4583-b2cd-4953-b5a0-642073169731","Type":"ContainerStarted","Data":"720a0ebae072a8d2bd47a6f37c76fb589ec26f83ed6481790cc564aa464c9606"}
Nov 03 00:52:52.602249 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:52.601941    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" podStartSLOduration=1.601923557 podStartE2EDuration="1.601923557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:52.594536849 +0000 UTC m=+717.808280522" watchObservedRunningTime="2026-11-03 00:52:52.601923557 +0000 UTC m=+717.815667229"
Nov 03 00:52:53.348184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:53.347725    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:52:53.806236 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:53.804702    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:52:53.806236 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:52:53.804765    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:00.194920 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.194878    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 00:53:00.199133 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.199112    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.205037 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.205010    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f7680d-openshift-default-dockercfg-fr542\""
Nov 03 00:53:00.212223 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.212197    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 00:53:00.321545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321715 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321559    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321715 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321691    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlcr4\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321789 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321823 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321793    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321857 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321857 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321845    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.321980 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.321968    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.322042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.322031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423395 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423350    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlcr4\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423599 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423407    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423599 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423434    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423599 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423628    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423764 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423927 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423781    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423927 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423790    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.423927 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423868    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.424095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423866    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.424095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.423991    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.424095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.424075    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.424522 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.424501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.425943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.425920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.426217 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.426191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.431043 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.431021    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.431131 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.431050    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlcr4\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.513809 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.513717    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:00.736763 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.736720    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 00:53:00.743008 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.742980    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.757559 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.757523    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 00:53:00.805368 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.805341    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 00:53:00.807919 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:00.807891    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535112b4_592c_4e5d_93a0_3628c871f421.slice/crio-52e72a3acafb801542812bf594444c660c9f767e19ea5a61123e5ff4b725e61b WatchSource:0}: Error finding container 52e72a3acafb801542812bf594444c660c9f767e19ea5a61123e5ff4b725e61b: Status 404 returned error can't find the container with id 52e72a3acafb801542812bf594444c660c9f767e19ea5a61123e5ff4b725e61b
Nov 03 00:53:00.810410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.810366    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:00.810488 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.810459    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:00.810526 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.810505    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827300    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827340    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827499    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827528    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhcr\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827565    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827606    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827677    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.828198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.827736    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928621 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928592    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928629    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928708    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928736    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhcr\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928766    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.928811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928803    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928843    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928869    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929088 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.928924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929263 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.929111    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929352 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.929328    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.929436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.929417    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.930234 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.930151    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.930361 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.930234    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.932358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.932332    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.933209 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.933154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.936428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.936402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:00.936623 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:00.936604    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhcr\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:01.068471 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.068383    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:01.348055 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.348018    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:53:01.357334 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.357307    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 00:53:01.357502 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:01.357313    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4163426a_911a_44e4_b50d_f30c2fe24131.slice/crio-c82b73d1a750aa2a9cdaee1722d53d8895dc970ac3286074eedf2690dac3229e WatchSource:0}: Error finding container c82b73d1a750aa2a9cdaee1722d53d8895dc970ac3286074eedf2690dac3229e: Status 404 returned error can't find the container with id c82b73d1a750aa2a9cdaee1722d53d8895dc970ac3286074eedf2690dac3229e
Nov 03 00:53:01.359988 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.359958    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:01.360093 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.360016    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:01.360093 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.360046    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:01.364842 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.364817    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:53:01.680728 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.680649    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" event={"ID":"535112b4-592c-4e5d-93a0-3628c871f421","Type":"ContainerStarted","Data":"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"}
Nov 03 00:53:01.680872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.680730    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" event={"ID":"535112b4-592c-4e5d-93a0-3628c871f421","Type":"ContainerStarted","Data":"52e72a3acafb801542812bf594444c660c9f767e19ea5a61123e5ff4b725e61b"}
Nov 03 00:53:01.684583 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.684485    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" event={"ID":"4163426a-911a-44e4-b50d-f30c2fe24131","Type":"ContainerStarted","Data":"7655805a17089be38bddf0cd55a5baed1b2c125c8d2f3ecc72fdb93e3d07ab69"}
Nov 03 00:53:01.684583 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.684548    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" event={"ID":"4163426a-911a-44e4-b50d-f30c2fe24131","Type":"ContainerStarted","Data":"c82b73d1a750aa2a9cdaee1722d53d8895dc970ac3286074eedf2690dac3229e"}
Nov 03 00:53:01.696667 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.696640    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:53:01.701388 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.701340    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podStartSLOduration=1.701324589 podStartE2EDuration="1.701324589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:01.700728261 +0000 UTC m=+726.914471939" watchObservedRunningTime="2026-11-03 00:53:01.701324589 +0000 UTC m=+726.915068262"
Nov 03 00:53:01.753186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:01.750203    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podStartSLOduration=1.750181091 podStartE2EDuration="1.750181091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:01.723867645 +0000 UTC m=+726.937611319" watchObservedRunningTime="2026-11-03 00:53:01.750181091 +0000 UTC m=+726.963924760"
Nov 03 00:53:02.074930 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.074141    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:02.075544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.075241    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:02.075544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.075297    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:02.514290 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.514206    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:02.515633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.515607    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:02.515742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:02.515660    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:03.070574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:03.070535    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:03.070866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:03.070834    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:03.514354 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:03.514280    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:03.514788 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:03.514754    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:04.075184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:04.074663    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:04.075184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:04.074745    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:04.514634 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:04.514541    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:04.515183 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:04.515134    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:05.072185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:05.071587    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:05.072185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:05.071667    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:05.517704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:05.517608    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:05.518337 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:05.518305    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:06.073701 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.073660    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:06.074025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.073990    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:06.165322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.165280    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:53:06.165598 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.165568    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:06.582232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.578357    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:06.582232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:06.578421    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:07.091188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:07.088820    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:07.091188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:07.088874    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:07.549245 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:07.545376    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:07.576182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:07.553387    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:08.076194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.075390    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:08.076194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.075468    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:08.542242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.539341    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:08.542242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.539399    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:08.918781 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.918743    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:53:08.919060 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:08.919036    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:09.079834 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:09.079797    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:09.080450 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:09.080415    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:09.515495 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:09.515452    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:53:09.515693 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:09.515516    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:53:09.882484 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:53:09.882451    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:53:10.069266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:10.069215    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:53:10.069442 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:10.069286    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:53:11.075383 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.075349    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:11.076213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.076190    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:11.077406 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.077385    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 00:53:11.335100 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.335028    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:11.335502 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.335476    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:11.336269 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.336248    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 00:53:11.527398 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.527365    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:53:11.883058 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.883008    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 00:53:11.887596 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.887571    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 00:53:11.887745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:11.887665    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014237 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014197    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqwj\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014247    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014337    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014400    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014437    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014660 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014673    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.014720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.014714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.116339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116304    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqwj\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116432    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116487    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116607    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116703    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116781    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.116903    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117680 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.117653    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.117700    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.117980 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.117961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.118036 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.118005    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.122042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.122014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.122850 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.122819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.125737 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.125712    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.126094 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.126072    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqwj\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.204428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.204342    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:12.487477 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.487395    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 00:53:12.492839 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:12.492800    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39ee48f_f016_4342_ae50_eb887c944279.slice/crio-ca4dc557aeeadb011f9324add7b24006487604c37e3a5c586bb1942be666c602 WatchSource:0}: Error finding container ca4dc557aeeadb011f9324add7b24006487604c37e3a5c586bb1942be666c602: Status 404 returned error can't find the container with id ca4dc557aeeadb011f9324add7b24006487604c37e3a5c586bb1942be666c602
Nov 03 00:53:12.495736 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.495703    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:12.495826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.495777    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:12.495826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:12.495807    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:13.067257 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:13.064230    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" event={"ID":"e39ee48f-f016-4342-ae50-eb887c944279","Type":"ContainerStarted","Data":"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"}
Nov 03 00:53:13.067257 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:13.064307    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" event={"ID":"e39ee48f-f016-4342-ae50-eb887c944279","Type":"ContainerStarted","Data":"ca4dc557aeeadb011f9324add7b24006487604c37e3a5c586bb1942be666c602"}
Nov 03 00:53:13.204798 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:13.204755    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:13.206700 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:13.206666    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:13.206854 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:13.206718    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:14.204750 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:14.204719    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:14.204928 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:14.204770    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:15.204773 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.204735    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:15.205093 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.205067    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:15.249433 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.249371    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podStartSLOduration=4.249348355 podStartE2EDuration="4.249348355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:13.089417014 +0000 UTC m=+738.303160687" watchObservedRunningTime="2026-11-03 00:53:15.249348355 +0000 UTC m=+740.463092029"
Nov 03 00:53:15.252936 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.252910    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 00:53:15.298281 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.298251    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 00:53:15.298506 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.298437    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.485921 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.485822    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j857d\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486101 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.485979    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486101 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486116    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486146    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486224    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486348 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486348 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486335    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.486425 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.486412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587195    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587252    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587416 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587323    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.587866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.587749    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.588655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.588030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.588655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.588081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.588655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.588115    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j857d\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.588655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.588239    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.589014 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.588989    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.589462 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.589421    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.589565 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.589468    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.592009 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.591374    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.593404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.593381    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.595848 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.595819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.597364 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.597341    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j857d\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:15.619220 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:15.618377    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:16.022668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.022621    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 00:53:16.023327 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:16.023250    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4d203_a9f5_4ffe_8e40_fd903d42f564.slice/crio-51dc905378e8ebfd6f0a125b19a5312a585a6d3533609d0b9d352529438cbb35 WatchSource:0}: Error finding container 51dc905378e8ebfd6f0a125b19a5312a585a6d3533609d0b9d352529438cbb35: Status 404 returned error can't find the container with id 51dc905378e8ebfd6f0a125b19a5312a585a6d3533609d0b9d352529438cbb35
Nov 03 00:53:16.026038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.026005    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:16.026107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.026071    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:16.026107 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.026100    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:16.093342 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.092247    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" event={"ID":"9bb4d203-a9f5-4ffe-8e40-fd903d42f564","Type":"ContainerStarted","Data":"51dc905378e8ebfd6f0a125b19a5312a585a6d3533609d0b9d352529438cbb35"}
Nov 03 00:53:16.204818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.204789    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:16.205041 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:16.204993    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:17.102892 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.102095    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" event={"ID":"9bb4d203-a9f5-4ffe-8e40-fd903d42f564","Type":"ContainerStarted","Data":"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"}
Nov 03 00:53:17.162399 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.162337    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podStartSLOduration=2.162317774 podStartE2EDuration="2.162317774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:17.159452454 +0000 UTC m=+742.373196129" watchObservedRunningTime="2026-11-03 00:53:17.162317774 +0000 UTC m=+742.376061447"
Nov 03 00:53:17.209354 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.209315    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:17.209532 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.209377    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:17.619567 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.619350    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:17.620699 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.620668    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:17.620793 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:17.620726    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:18.205143 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:18.205099    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:18.205804 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:18.205765    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:18.619061 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:18.619029    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:18.619250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:18.619083    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:19.205320 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:19.205281    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:53:19.205906 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:19.205874    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 00:53:19.619747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:19.619697    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:19.619965 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:19.619767    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:20.620633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:20.620593    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:20.621277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:20.621241    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:21.250899 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.250260    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:53:21.250899 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.250326    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:21.619113 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.619082    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:21.619339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.619137    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:21.996667 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.996592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:21.997089 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.996912    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:21.997799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:21.997779    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 00:53:22.619342 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:22.619303    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:53:22.619522 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:22.619360    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:53:23.764755 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:23.764718    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:23.765937 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:23.765835    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:23.767275 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:23.767239    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 00:53:26.555246 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:26.555205    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" podUID="330f4583-b2cd-4953-b5a0-642073169731" containerName="main" containerID="cri-o://0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1" gracePeriod=45
Nov 03 00:53:26.685625 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:26.685586    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" podUID="330f4583-b2cd-4953-b5a0-642073169731" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Nov 03 00:53:26.868888 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:26.868851    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:53:27.123705 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.123887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123714    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\" (UniqueName: \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.123887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123767    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.123887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123824    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.123887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123858    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\" (UniqueName: \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.124102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\") pod \"330f4583-b2cd-4953-b5a0-642073169731\" (UID: \"330f4583-b2cd-4953-b5a0-642073169731\") "
Nov 03 00:53:27.124102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.123929    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:27.124265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.124187    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:27.124265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.124219    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:27.124955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.124875    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.124955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.124915    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.124955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.124930    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.126801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.126766    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:53:27.126801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.126781    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:27.126936 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.126798    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj" pod "330f4583-b2cd-4953-b5a0-642073169731" (UID: "330f4583-b2cd-4953-b5a0-642073169731"). InnerVolumeSpecName "kube-api-access-2q4zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:53:27.209965 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.209928    4531 generic.go:356] "Generic (PLEG): container finished" podID="330f4583-b2cd-4953-b5a0-642073169731" containerID="0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1" exitCode=0
Nov 03 00:53:27.210204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.210017    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"
Nov 03 00:53:27.210204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.210088    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" event={"ID":"330f4583-b2cd-4953-b5a0-642073169731","Type":"ContainerDied","Data":"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"}
Nov 03 00:53:27.210204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.210125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn" event={"ID":"330f4583-b2cd-4953-b5a0-642073169731","Type":"ContainerDied","Data":"720a0ebae072a8d2bd47a6f37c76fb589ec26f83ed6481790cc564aa464c9606"}
Nov 03 00:53:27.210204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.210145    4531 scope.go:122] "RemoveContainer" containerID="0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"
Nov 03 00:53:27.225352 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:53:27.225323    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod330f4583_b2cd_4953_b5a0_642073169731.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 00:53:27.226285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.226210    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/330f4583-b2cd-4953-b5a0-642073169731-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.226285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.226235    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/330f4583-b2cd-4953-b5a0-642073169731-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.226285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.226250    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4zj\" (UniqueName: \"kubernetes.io/projected/330f4583-b2cd-4953-b5a0-642073169731-kube-api-access-2q4zj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:53:27.233497 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.233462    4531 scope.go:122] "RemoveContainer" containerID="0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"
Nov 03 00:53:27.233847 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:53:27.233819    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1\": container with ID starting with 0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1 not found: ID does not exist" containerID="0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"
Nov 03 00:53:27.233973 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.233854    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1"} err="failed to get container status \"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1\": rpc error: code = NotFound desc = could not find container \"0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1\": container with ID starting with 0337e95351799e3d8abbeb8d5c012b9126e066e6f0b30dd18b203e277c921fe1 not found: ID does not exist"
Nov 03 00:53:27.241309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.241281    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:53:27.246814 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.246792    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/e2e-conflict-b-45480c9d-kserve-565d8fbd44-t8vxn"]
Nov 03 00:53:27.672177 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.672120    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 00:53:27.677932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.677898    4531 cpu_manager.go:414] "containerMap: removing container" podUID="330f4583-b2cd-4953-b5a0-642073169731" containerName="main"
Nov 03 00:53:27.677932 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.677934    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330f4583-b2cd-4953-b5a0-642073169731" containerName="main"
Nov 03 00:53:27.678244 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.678141    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="330f4583-b2cd-4953-b5a0-642073169731" containerName="main"
Nov 03 00:53:27.683436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.683410    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 00:53:27.683615 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.683513    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.694719 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.692881    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"openshift-service-ca.crt\""
Nov 03 00:53:27.694719 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.693030    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"default-dockercfg-ggdgp\""
Nov 03 00:53:27.694719 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.693227    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"kube-root-ca.crt\""
Nov 03 00:53:27.694719 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.693362    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"distinct-w4-650be0-kserve-self-signed-certs\""
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.874799    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.874888    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.875010    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.875040    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.875064    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstqt\" (UniqueName: \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.875221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.875100    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976465 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976362    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976465 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976419    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976465 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976450    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstqt\" (UniqueName: \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976482    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976546    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.976998 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.976976    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.977529 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.977502    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.977691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.977520    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.977820 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.977531    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.979503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.979478    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.979773 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.979750    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:27.985285 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:27.985262    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstqt\" (UniqueName: \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\") pod \"distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:28.003241 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:28.003212    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:28.287189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:28.287144    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 00:53:28.288678 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:28.288652    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e524439_cc34_4594_8570_64ba80a8ea8d.slice/crio-da4a83f29600a6349fdd91980203784a46d9fa8ac4e2bd3666aa45b358fa1a81 WatchSource:0}: Error finding container da4a83f29600a6349fdd91980203784a46d9fa8ac4e2bd3666aa45b358fa1a81: Status 404 returned error can't find the container with id da4a83f29600a6349fdd91980203784a46d9fa8ac4e2bd3666aa45b358fa1a81
Nov 03 00:53:28.903676 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:28.903642    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330f4583-b2cd-4953-b5a0-642073169731" path="/var/lib/kubelet/pods/330f4583-b2cd-4953-b5a0-642073169731/volumes"
Nov 03 00:53:29.225377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:29.225292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" event={"ID":"0e524439-cc34-4594-8570-64ba80a8ea8d","Type":"ContainerStarted","Data":"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"}
Nov 03 00:53:29.225377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:29.225341    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" event={"ID":"0e524439-cc34-4594-8570-64ba80a8ea8d","Type":"ContainerStarted","Data":"da4a83f29600a6349fdd91980203784a46d9fa8ac4e2bd3666aa45b358fa1a81"}
Nov 03 00:53:29.241772 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:29.241722    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podStartSLOduration=2.241704827 podStartE2EDuration="2.241704827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:29.240860377 +0000 UTC m=+754.454604075" watchObservedRunningTime="2026-11-03 00:53:29.241704827 +0000 UTC m=+754.455448499"
Nov 03 00:53:30.003383 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:30.003346    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:38.003542 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:38.003451    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:38.016947 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:38.016920    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:38.312190 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:38.312090    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:39.878305 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:53:39.878267    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:53:45.830729 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:45.830686    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 00:53:45.837324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:45.837293    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016731    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016796    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016839    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsvr\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016896    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.016974    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.017005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.019184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.017041    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.033651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.026559    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 00:53:46.129922 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.129820    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.130302 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.130274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.131610 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.131583    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.131770 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.131750    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsvr\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.131899 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.131880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.132011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.131996    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.132208 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.132190    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.132357 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.132339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.132483 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.132468    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.133063 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.130607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.133655 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.133633    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.133784 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.131202    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.134517 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.134495    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.135595 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.135543    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.148547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.148518    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.149627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.149565    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.161219 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.157840    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsvr\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.170194 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.169683    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\") pod \"e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") " pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:46.341457 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:46.341062    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:47.334186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.331245    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:53:47.454709 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.435608    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 00:53:47.503186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.484661    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:47.503186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.484753    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:47.503186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.484799    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:53:47.792186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.791691    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:47.891239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:47.882308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:48.335769 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.335720    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:48.336958 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.336933    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:48.500544 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.500439    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:53:48.558769 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.558733    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" event={"ID":"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e","Type":"ContainerStarted","Data":"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"}
Nov 03 00:53:48.559022 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.559001    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" event={"ID":"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e","Type":"ContainerStarted","Data":"e51efe69547a314556f148e26cfeb667ccf2be6188093e1b9709393d291389e2"}
Nov 03 00:53:48.592617 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:48.592483    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podStartSLOduration=3.5924623589999998 podStartE2EDuration="3.592462359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:48.588070908 +0000 UTC m=+773.801814588" watchObservedRunningTime="2026-11-03 00:53:48.592462359 +0000 UTC m=+773.806206031"
Nov 03 00:53:49.348759 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:49.347183    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:49.361343 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:49.361311    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:49.361627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:49.361580    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:50.344778 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:50.344742    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:50.345106 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:50.345057    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:51.348819 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:51.346601    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:51.348819 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:51.346678    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:52.347186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:52.345584    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:52.347186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:52.345648    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:53.348315 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:53.348269    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:53.348917 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:53.348880    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:54.005870 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.004059    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 00:53:54.008222 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.008200    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.018213 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.017707    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-650be0\"/\"distinct-2-w4-650be0-kserve-self-signed-certs\""
Nov 03 00:53:54.022218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.022188    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 00:53:54.115522 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.115717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115615    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.115717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115693    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.115956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115741    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.115956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115764    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jq4d\" (UniqueName: \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.115956 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.115803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.217886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.217834    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.218999 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.218596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.219227 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.219206    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jq4d\" (UniqueName: \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.219758 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.219737    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.220321 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.220301    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.220827 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.220706    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.221046 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.221027    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.236051 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.236021    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.239077 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.221504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.245845 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.245807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.246341 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.246321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.290236 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.281601    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jq4d\" (UniqueName: \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\") pod \"distinct-2-w4-650be0-kserve-8456c84f5b-68dkn\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") " pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.346603 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.346564    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:54.346915 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.346875    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:54.364590 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.364559    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:54.702122 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:54.702089    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 00:53:54.702878 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:53:54.702848    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf037a05a_daaf_4185_9405_2fd5e4112f6a.slice/crio-23dad678fef0f6ff4e1ce4dee9eadb654d47e8d1a3e2c45a211c3b24826b281f WatchSource:0}: Error finding container 23dad678fef0f6ff4e1ce4dee9eadb654d47e8d1a3e2c45a211c3b24826b281f: Status 404 returned error can't find the container with id 23dad678fef0f6ff4e1ce4dee9eadb654d47e8d1a3e2c45a211c3b24826b281f
Nov 03 00:53:55.342901 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:55.342861    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:53:55.343095 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:55.342930    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 00:53:55.615038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:55.614942    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" event={"ID":"f037a05a-daaf-4185-9405-2fd5e4112f6a","Type":"ContainerStarted","Data":"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"}
Nov 03 00:53:55.615038 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:55.614983    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" event={"ID":"f037a05a-daaf-4185-9405-2fd5e4112f6a","Type":"ContainerStarted","Data":"23dad678fef0f6ff4e1ce4dee9eadb654d47e8d1a3e2c45a211c3b24826b281f"}
Nov 03 00:53:55.634465 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:55.634415    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podStartSLOduration=2.6343997249999997 podStartE2EDuration="2.634399725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:55.632241514 +0000 UTC m=+780.845985189" watchObservedRunningTime="2026-11-03 00:53:55.634399725 +0000 UTC m=+780.848143396"
Nov 03 00:53:56.365812 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:56.365775    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:53:56.838328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:56.838291    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:56.839215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:56.839192    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:53:56.839774 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:53:56.839754    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 00:54:00.326914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.326865    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 00:54:00.333504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.333472    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.346976 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.346946    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 00:54:00.423155 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.423119    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 00:54:00.432482 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.432456    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.438814 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.438788    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2f9a3a\"/\"distinct-2-w0-2f9a3a-kserve-self-signed-certs\""
Nov 03 00:54:00.441822 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.441796    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 00:54:00.448230 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448195    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448427 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448497    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448571    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448595    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkt5t\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.448890 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.448837    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.549895 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.549840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.549895 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.549898    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.549989    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550116    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550183    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550209    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550283 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550235    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkt5t\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550453 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550453 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550402    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb5h4\" (UniqueName: \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550564 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550482    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550564 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550537    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550572    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550645    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550775 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550695    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.550775 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550733    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.550950 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550928    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.551013 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.550977    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.551388 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.551359    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.551504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.551409    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.551574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.551530    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.553867 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.553845    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.560975 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.560919    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkt5t\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.563517 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.563465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.581861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.581745    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.651852    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.651915    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.651948    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.651973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.652049    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.652191    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb5h4\" (UniqueName: \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.652409    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.652923    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.655231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.653070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.656481 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.656418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.664820 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.664442    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.664820 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.664773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb5h4\" (UniqueName: \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\") pod \"distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") " pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.675271 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.674859    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:00.764182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.760874    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:00.975840 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.975811    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 00:54:00.977359 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:54:00.977322    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a344b50_a4c2_4d77_bfce_1756c0ba8d4a.slice/crio-58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd WatchSource:0}: Error finding container 58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd: Status 404 returned error can't find the container with id 58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd
Nov 03 00:54:00.980264 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.980221    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:00.980361 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.980317    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:00.980431 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:00.980362    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:01.075706 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.075675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 00:54:01.078257 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:54:01.078228    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod675f003d_3ddc_4e7f_9743_00e7119b1ab5.slice/crio-408ce894a580cac0c4f958c37ce3a5683149fd606500e16f525cd7a038b4398c WatchSource:0}: Error finding container 408ce894a580cac0c4f958c37ce3a5683149fd606500e16f525cd7a038b4398c: Status 404 returned error can't find the container with id 408ce894a580cac0c4f958c37ce3a5683149fd606500e16f525cd7a038b4398c
Nov 03 00:54:01.711653 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.711597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" event={"ID":"675f003d-3ddc-4e7f-9743-00e7119b1ab5","Type":"ContainerStarted","Data":"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"}
Nov 03 00:54:01.712294 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.712253    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" event={"ID":"675f003d-3ddc-4e7f-9743-00e7119b1ab5","Type":"ContainerStarted","Data":"408ce894a580cac0c4f958c37ce3a5683149fd606500e16f525cd7a038b4398c"}
Nov 03 00:54:01.716358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.716331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" event={"ID":"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a","Type":"ContainerStarted","Data":"ce4ec99b201c5f4323b55fe021a6a2e7787690d6aea1310480509a90d230bb64"}
Nov 03 00:54:01.716573 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.716557    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" event={"ID":"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a","Type":"ContainerStarted","Data":"58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd"}
Nov 03 00:54:01.741182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.738428    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podStartSLOduration=1.738409382 podStartE2EDuration="1.738409382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:01.737574188 +0000 UTC m=+786.951317862" watchObservedRunningTime="2026-11-03 00:54:01.738409382 +0000 UTC m=+786.952153056"
Nov 03 00:54:01.770725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:01.770591    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:02.696185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:02.685376    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:02.696185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:02.685518    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:02.696185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:02.685563    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:03.678184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:03.677578    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:03.678184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:03.677641    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:04.370336 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:04.370299    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:54:04.683497 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:04.683409    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:04.683497 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:04.683470    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:05.386791 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:05.386726    4531 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 00:54:05.445666 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:05.445606    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:05.505251 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:05.505206    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:05.675897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:05.675819    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:05.675897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:05.675872    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:06.675589 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:06.675547    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:06.676084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:06.675607    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:07.676171 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:07.676126    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:07.676604 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:07.676204    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:08.675635 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:08.675595    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:08.675839 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:08.675652    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:09.675869 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:09.675832    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:54:09.676260 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:09.675887    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:54:09.882466 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:54:09.882427    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:54:10.680744 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.680715    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:10.681238 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.681140    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:10.682035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.682009    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 00:54:10.703206 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.703129    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podStartSLOduration=10.703117756 podStartE2EDuration="10.703117756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:01.770487562 +0000 UTC m=+786.984231236" watchObservedRunningTime="2026-11-03 00:54:10.703117756 +0000 UTC m=+795.916861428"
Nov 03 00:54:10.761901 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.761853    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:10.774988 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.774960    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:10.912195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:10.912153    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:54:11.944211 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:11.944180    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 00:54:11.948291 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:11.948265    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:11.966723 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:11.966689    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 00:54:12.092216 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092179    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092216 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092220    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092264    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4s2l\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092474 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092436    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092585 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092528    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092749 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092726    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092769    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092889 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092871    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.092958 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.092945    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.193633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193601    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.193821 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193640    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.193821 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193743    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.193821 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193816    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193859    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.193981    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s2l\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194059 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194070    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194508    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.194560 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.194560    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.196420 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.196392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.196607 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.196586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.201649 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.201624    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s2l\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.201729 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.201703    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\") pod \"e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") " pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.267917 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.267881    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:12.568842 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.568808    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 00:54:12.569698 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:54:12.569672    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad49df2b_f905_40c4_aa39_cf966d5dbc25.slice/crio-8f90f345bfe89ed887a6b7e100d028edde17439d27596a355b0109cd3b561b47 WatchSource:0}: Error finding container 8f90f345bfe89ed887a6b7e100d028edde17439d27596a355b0109cd3b561b47: Status 404 returned error can't find the container with id 8f90f345bfe89ed887a6b7e100d028edde17439d27596a355b0109cd3b561b47
Nov 03 00:54:12.572250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.572217    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:12.572326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.572287    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:12.572326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.572319    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:12.915105 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.915059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" event={"ID":"ad49df2b-f905-40c4-aa39-cf966d5dbc25","Type":"ContainerStarted","Data":"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"}
Nov 03 00:54:12.915303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.915122    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" event={"ID":"ad49df2b-f905-40c4-aa39-cf966d5dbc25","Type":"ContainerStarted","Data":"8f90f345bfe89ed887a6b7e100d028edde17439d27596a355b0109cd3b561b47"}
Nov 03 00:54:12.942522 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:12.942438    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podStartSLOduration=1.942423769 podStartE2EDuration="1.942423769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:12.938894872 +0000 UTC m=+798.152638548" watchObservedRunningTime="2026-11-03 00:54:12.942423769 +0000 UTC m=+798.156167477"
Nov 03 00:54:13.268547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:13.268455    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:13.269985 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:13.269957    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:13.270135 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:13.270016    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:14.268983 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:14.268944    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:14.269392 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:14.269013    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:14.377509 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:14.377478    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:54:14.389711 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:14.389688    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:54:15.268401 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:15.268362    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:15.268597 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:15.268431    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:16.269115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:16.269069    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:16.269603 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:16.269140    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:17.268542 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:17.268505    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:17.268801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:17.268559    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:18.268999 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:18.268960    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:18.269532 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:18.269081    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:19.269611 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:19.269571    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:54:19.270269 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:19.270237    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:54:20.844712 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:20.844678    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:20.845233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:20.845207    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:20.846964 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:20.846945    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 00:54:21.984412 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:21.984375    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 00:54:21.991249 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:21.991219    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:21.998738 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:21.998710    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 00:54:21.999373 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:21.999145    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-88bbf7\"/\"distinct-w1-88bbf7-kserve-self-signed-certs\""
Nov 03 00:54:22.036489 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036455    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.036751 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036720    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.036890 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mwx\" (UniqueName: \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.036890 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036848    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.037009 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036919    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.037009 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.036986    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.138749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.138840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.138867    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.138947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mwx\" (UniqueName: \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.139001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.139111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.139909    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.140148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.140415 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.140416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.145347 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.145320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.145812 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.145784    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.158791 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.158759    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mwx\" (UniqueName: \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\") pod \"distinct-w1-88bbf7-kserve-64754949f9-jlvvm\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") " pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.323246 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.323213    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:22.626326 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:22.626292    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 00:54:22.627459 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:54:22.627430    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f203175_33d7_44d8_9062_e5e592473a97.slice/crio-8385b5ff185ddece9966e992ef9a63f8b115054cc1664904cdc78070aa2e78c9 WatchSource:0}: Error finding container 8385b5ff185ddece9966e992ef9a63f8b115054cc1664904cdc78070aa2e78c9: Status 404 returned error can't find the container with id 8385b5ff185ddece9966e992ef9a63f8b115054cc1664904cdc78070aa2e78c9
Nov 03 00:54:23.049943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:23.049837    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" event={"ID":"9f203175-33d7-44d8-9062-e5e592473a97","Type":"ContainerStarted","Data":"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"}
Nov 03 00:54:23.049943 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:23.049921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" event={"ID":"9f203175-33d7-44d8-9062-e5e592473a97","Type":"ContainerStarted","Data":"8385b5ff185ddece9966e992ef9a63f8b115054cc1664904cdc78070aa2e78c9"}
Nov 03 00:54:23.069007 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:23.068954    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podStartSLOduration=2.068937242 podStartE2EDuration="2.068937242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:23.066743057 +0000 UTC m=+808.280486730" watchObservedRunningTime="2026-11-03 00:54:23.068937242 +0000 UTC m=+808.282680915"
Nov 03 00:54:23.324031 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:23.324000    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:31.863565 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:31.863520    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 00:54:32.439948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.439893    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:54:32.494192 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.445645    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:32.537182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.533387    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 00:54:32.669185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.667406    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:32.669185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.667455    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 00:54:32.669185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.667562    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.695462 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.695371    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2f9a3a-openshift-default-dockercfg-gdpt5\""
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777745    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777836    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777877    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777936    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.777988    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsz6\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.783501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.778068    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:32.823977 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.823915    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:32.986745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:32.986630    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:33.157985 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.157947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.163125 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.163090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.163636 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.163606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.167256 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.167229    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.167448 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.167430    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.169704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.169681    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.170125 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.170102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.172294 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.172262    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsz6\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.173238 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.173211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.208175 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.206923    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.210047 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.210014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.212491 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.212467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.217736 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.216387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.217736 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.216690    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.217736 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.216980    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.220189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.220094    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.223941 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.223450    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.228212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.226517    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsz6\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.229103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.228755    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:33.810683 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.810591    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:54:33.886766 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:33.870911    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:34.373070 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.373035    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 00:54:34.386419 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.386380    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:34.386551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.386467    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:34.386551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.386513    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:34.494945 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.494905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" event={"ID":"e3c40d84-4cde-4104-84c5-cb356da785e4","Type":"ContainerStarted","Data":"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"}
Nov 03 00:54:34.495103 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:34.494978    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" event={"ID":"e3c40d84-4cde-4104-84c5-cb356da785e4","Type":"ContainerStarted","Data":"15c4bcbd9e777bdef3e6dcf7f01fa7b1009c39165a71a1811dbce2be84533532"}
Nov 03 00:54:35.263507 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:35.263300    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:35.264868 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:35.264838    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:35.264987 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:35.264895    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:36.264187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:36.263841    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:36.264187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:36.263904    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:37.263087 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:37.263046    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:37.263346 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:37.263107    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:38.263715 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:38.263679    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:38.264097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:38.263737    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:39.263691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:39.263654    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:39.263873 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:39.263712    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:39.886548 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:54:39.886511    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:54:40.263688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:40.263599    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:40.263688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:40.263656    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:41.263482 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:41.263439    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 00:54:41.263887 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:41.263507    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 00:54:42.269362 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:42.269326    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:42.269844 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:42.269735    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:42.270640 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:42.270623    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 00:54:42.291581 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:42.291529    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podStartSLOduration=10.291515956 podStartE2EDuration="10.291515956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:34.515270707 +0000 UTC m=+819.729014379" watchObservedRunningTime="2026-11-03 00:54:42.291515956 +0000 UTC m=+827.505259636"
Nov 03 00:54:47.511184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.510418    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 00:54:47.526510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.526480    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.548898 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.535967    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 00:54:47.612298 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.577065    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.613140 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.613115    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.613342 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.613326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.622409 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.622378    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.644198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.644025    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.644399 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.644382    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.644631 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.644613    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvf9h\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.644771 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.644757    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.664250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.664209    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814004 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.813907    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814324 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814299    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814464 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814447    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814578 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvf9h\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814700 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814685    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814808 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814794    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.814994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.814977    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.815254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.815232    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.815413 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.815393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.818183 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.816035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.819923 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.819898    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.820314 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.820295    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.820978 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.820958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.840188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.830150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.841242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.840674    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.856939 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.856859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvf9h\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.869187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.859374    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:47.883813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:47.883747    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:48.130186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:48.120275    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:49.408516 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:49.408481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 00:54:49.504481 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:49.504422    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:54:49.562601 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:49.562527    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:49.562740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:49.562618    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:49.562740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:49.562660    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:50.086232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.086193    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" event={"ID":"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba","Type":"ContainerStarted","Data":"153f11294e8bfbd5afea776f4c13ce8c9835d136416387803f61596d25e9ff77"}
Nov 03 00:54:50.086408 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.086262    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" event={"ID":"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba","Type":"ContainerStarted","Data":"da0bcbc5b14d2fec1d30b4933b7caa1c8bf2c4f27c1fca1e3d5554055e9ecd62"}
Nov 03 00:54:50.114523 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.114475    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podStartSLOduration=3.114461435 podStartE2EDuration="3.114461435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:50.110530168 +0000 UTC m=+835.324273849" watchObservedRunningTime="2026-11-03 00:54:50.114461435 +0000 UTC m=+835.328205106"
Nov 03 00:54:50.122439 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.122407    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:50.124118 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.124081    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:50.124738 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:50.124450    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:51.121978 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:51.121916    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:51.122518 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:51.122002    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:52.122547 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:52.122503    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:52.123023 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:52.122568    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:53.121740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.121708    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:53.122005 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.121977    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:53.585436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.585404    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 00:54:53.589801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.589781    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.596274 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.596252    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-88bbf7-openshift-default-dockercfg-h6n25\""
Nov 03 00:54:53.603226 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.603203    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.608972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609074    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609092    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609118    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609187    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9hx\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609227    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609249    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.610279 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.609336    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710074 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710250    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710312    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710361    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710507 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710443    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710507 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710507 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710500    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710631 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710631 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710590    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9hx\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.710865 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710842    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.711014 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710872    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.711104 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.710928    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.711144 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.711118    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.713025 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.713000    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.713378 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.713356    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.718642 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.718619    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.718793 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.718776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9hx\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:53.905753 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:53.905677    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:54.121660 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.121620    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:54.121830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.121688    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:54.257513 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.244087    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:54.257513 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.244200    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:54.257513 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.244250    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:54:54.266225 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:54.265061    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 00:54:55.122152 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.122114    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:55.122653 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.122209    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:55.125320 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.125214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" event={"ID":"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2","Type":"ContainerStarted","Data":"08ea36b41483f1fc91bc3a0612c8d9f5190b53ce11d9a1b435a7be5a0361f72f"}
Nov 03 00:54:55.125470 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.125322    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" event={"ID":"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2","Type":"ContainerStarted","Data":"49e63f212af77b38fe19a8f0db7ff6402eecc35185fda0049608a052c35e69ed"}
Nov 03 00:54:55.176008 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.175958    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podStartSLOduration=2.175943827 podStartE2EDuration="2.175943827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:55.175390967 +0000 UTC m=+840.389134640" watchObservedRunningTime="2026-11-03 00:54:55.175943827 +0000 UTC m=+840.389687499"
Nov 03 00:54:55.906455 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.906414    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:54:55.908242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.908216    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:54:55.908389 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:55.908272    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:54:56.122346 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:56.122313    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:54:56.122744 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:56.122367    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:54:56.906584 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:56.906549    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:54:56.906806 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:56.906600    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:54:57.358187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:57.349550    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:57.358187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:57.351065    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:57.358187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:57.353614    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 00:54:57.906634 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:57.906596    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:54:57.906850 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:57.906656    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:54:58.906020 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:58.905987    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:54:58.906451 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:58.906068    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:54:59.906852 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:59.906812    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:54:59.907235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:54:59.906886    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:55:00.912197 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:00.912143    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:55:00.912768 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:00.912738    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:55:01.916692 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:01.916657    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:55:01.917362 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:01.917316    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:55:02.910185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:02.909026    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 00:55:02.910185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:02.909087    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 00:55:03.568185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:03.568118    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:03.588183 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:03.568744    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:03.901218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:03.901143    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:04.045186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:03.922730    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:04.070186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.047832    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 00:55:04.599196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.576251    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:04.599196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.577827    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:55:04.658696 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.658573    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:04.669191 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.659573    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:55:04.813802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.813759    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:04.972707 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.972609    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 00:55:04.993705 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:04.993657    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:05.046186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.045875    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:55:05.046186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.045935    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:05.048856 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.048828    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:55:05.049097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.049082    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:55:05.049241 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.049229    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:55:05.049358 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.049348    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:55:05.096182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.094456    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:55:05.107229 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:05.107017    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:55:06.087351 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:06.087299    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:55:06.087868 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:06.087355    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:06.489232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:06.489119    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:55:06.489683 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:06.489664    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:55:06.490427 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:06.490409    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 00:55:09.884671 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:55:09.884637    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:55:18.732138 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.732091    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:55:18.738385 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.738359    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.745512 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.745481    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd03e395-serving-cert\""
Nov 03 00:55:18.750686 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.750651    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:55:18.886117 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.886076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.886352 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.886326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwkf\" (UniqueName: \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.886428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.886404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.886534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.886518    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.988483 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.988388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.988671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.988495    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwkf\" (UniqueName: \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.988671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.988524    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.988671 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.988587    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.989395 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.989351    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.990966 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.990935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:18.991223 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:18.991194    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:19.005289 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:19.005252    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwkf\" (UniqueName: \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-q4d5x\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:19.069928 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:19.069890    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:19.411215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:19.402430    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:55:19.594203 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:19.592155    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" event={"ID":"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6","Type":"ContainerStarted","Data":"48851326c86bd89c530f1ab82c4ec5178c721cffd1c0a8dabec8a1e0fcc04b61"}
Nov 03 00:55:20.702397 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:20.702296    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" event={"ID":"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6","Type":"ContainerStarted","Data":"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"}
Nov 03 00:55:20.720186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:20.707301    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:20.760189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:20.759769    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" podStartSLOduration=2.047244865 podStartE2EDuration="2.759747051s" totalImagesPullingTime="712.502186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:20.758831963 +0000 UTC m=+865.972575646" watchObservedRunningTime="2026-11-03 00:55:20.759747051 +0000 UTC m=+865.973490724"
Nov 03 00:55:21.067251 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.065676    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:55:21.103460 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.102255    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:21.749727 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.749690    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:21.783187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.782247    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:21.797766 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.796512    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:21.838436 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:21.838363    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.151250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.147583    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.151250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.148056    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:55:22.151250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.148740    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:55:22.259187 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.252663    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.263047 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.259637    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:55:22.597897 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.597814    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:55:22.598118 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.597999    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.677186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.676203    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 00:55:22.787188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.785496    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.787188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.786238    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:55:22.859247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.801144    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:22.859247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.808903    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:55:22.859247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.839702    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 00:55:22.859247 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:22.853283    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:55:23.164813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.156983    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:23.164813 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.158211    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:55:23.299339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.260929    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:55:23.328108 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.328069    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:23.834730 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.834682    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:23.890617 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.865264    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:55:23.927311 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.927257    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:23.927613 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.899702    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:23.928084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:23.868001    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 00:55:24.136454 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.136363    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 00:55:24.212718 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.212670    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:24.214070 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.214037    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:24.261184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.258685    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 00:55:24.261184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.258826    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.359822 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.359783    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408573    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408742    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408777    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408834    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408869    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408921    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.415722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.408947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drkp5\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514110    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514137    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514185    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514230    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514259    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drkp5\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.518186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.514492    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.544184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.542061    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.544184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.542390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.544184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.542983    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.549872 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.549843    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.565185 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.553510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.611243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.611197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drkp5\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.620193 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.617112    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.620193 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.617633    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.620193 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.619825    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.707255 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.687280    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:55:24.711497 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.711225    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:24.941114 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.941059    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:24.956400 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.955640    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 00:55:24.965337 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:24.956811    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 00:55:25.226404 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.226315    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:55:25.247303 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.226658    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:25.387763 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.387713    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:25.465183 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.462589    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:25.701848 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.701814    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 00:55:25.912195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:25.905274    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 00:55:26.008366 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.008245    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:26.098522 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.089980    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 00:55:26.287233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.257123    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:26.287233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.282313    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:26.287233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.282408    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:26.287233 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.282451    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:26.377561 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.371608    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" event={"ID":"d4c35222-39b0-405f-8248-747f49e20090","Type":"ContainerStarted","Data":"75afa3fd91351ae1f588a2e13ea39752e8777f96ad6b1a8147ba91075fd7901e"}
Nov 03 00:55:26.457702 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.452231    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 00:55:26.481453 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.478880    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 00:55:26.482217 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:26.481762    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 00:55:27.385591 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:27.382548    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" event={"ID":"d4c35222-39b0-405f-8248-747f49e20090","Type":"ContainerStarted","Data":"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"}
Nov 03 00:55:27.405898 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:27.405847    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podStartSLOduration=4.4058326789999995 podStartE2EDuration="4.405832679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:27.403312595 +0000 UTC m=+872.617056306" watchObservedRunningTime="2026-11-03 00:55:27.405832679 +0000 UTC m=+872.619576350"
Nov 03 00:55:27.721820 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:27.721735    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:27.723006 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:27.722981    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:27.723125 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:27.723028    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:28.076188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:28.072536    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:55:28.721496 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:28.721460    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:28.722031 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:28.721522    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:29.720811 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:29.720773    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:29.721023 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:29.720836    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:30.721308 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:30.721273    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:30.721718 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:30.721327    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:31.721086 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:31.721029    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:31.721270 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:31.721113    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:32.655538 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.655501    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 00:55:32.660801 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.660775    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.721586 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.721542    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:55:32.721757 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.721615    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 00:55:32.767975 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.767930    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768119    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768196 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768473 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768213    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xf6\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768721    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.768914 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768895    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.769012 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.768956    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.769144 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.769118    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.788421 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.788385    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 00:55:32.870830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.870789    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.870830 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.870833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.870886    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xf6\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.870915    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.870958    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871004    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871084 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871075    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871401 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871358    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871582    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871594    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871822    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.871886 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.871872    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.872136 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.872110    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.873962 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.873937    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.874200 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.874182    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.879972 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.879944    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xf6\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.880054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.880033    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:32.977136 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:32.977051    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:33.275616 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.275580    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 00:55:33.279802 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:55:33.276755    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d188bfe_c7cd_48f5_be7d_9416a81b417c.slice/crio-227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea WatchSource:0}: Error finding container 227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea: Status 404 returned error can't find the container with id 227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea
Nov 03 00:55:33.279802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.279663    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:33.279802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.279725    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:33.279802 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.279754    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:33.432843 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.432792    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" event={"ID":"2d188bfe-c7cd-48f5-be7d-9416a81b417c","Type":"ContainerStarted","Data":"65abfb1c02afecb876507a843d04f8d4628e2b7ba00367b5129a3d21798725b1"}
Nov 03 00:55:33.433047 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.432871    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" event={"ID":"2d188bfe-c7cd-48f5-be7d-9416a81b417c","Type":"ContainerStarted","Data":"227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea"}
Nov 03 00:55:33.460002 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.459938    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podStartSLOduration=1.459915832 podStartE2EDuration="1.459915832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:33.452879334 +0000 UTC m=+878.666623007" watchObservedRunningTime="2026-11-03 00:55:33.459915832 +0000 UTC m=+878.673659504"
Nov 03 00:55:33.980182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:33.979269    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:34.291063 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:34.290986    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:34.291420 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:34.291399    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:34.292134 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:34.292115    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 00:55:34.978861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:34.978819    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:34.979046 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:34.978880    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:35.977846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:35.977806    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:35.978265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:35.977885    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:36.977800 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:36.977756    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:36.977994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:36.977835    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:37.977836 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:37.977799    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:37.978126 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:37.978097    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:38.977483 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:38.977442    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:38.977705 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:38.977519    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:39.881353 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:55:39.881313    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:55:39.978282 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:39.978247    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:55:39.978445 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:39.978306    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:55:40.982299 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:40.982268    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:40.982793 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:40.982658    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:40.983577 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:40.983553    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 00:55:46.486343 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:46.486292    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:46.989183 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:46.954547    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:55:47.304242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:47.274448    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 00:55:47.304242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:47.276261    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:55:47.304242 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:55:47.291321    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:56:06.920131 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:06.920095    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:06.926567 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:06.926542    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:06.947419 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:06.945537    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-51119c-serving-cert\""
Nov 03 00:56:06.952710 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:06.952664    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:07.006970 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.006932    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.007146 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.007104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.007254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.007224    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.007320 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.007285    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46jrr\" (UniqueName: \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.109607 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.109561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46jrr\" (UniqueName: \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.110651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.110324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.110651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.110478    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.110651 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.110577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.111498 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.111469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.115995 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.115958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.117257 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.117230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.119580 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.119551    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46jrr\" (UniqueName: \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\") pod \"maas-api-e2e-api-a-51119c-7d7689f997-pw2g4\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.269201 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.269062    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:07.561337 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:07.561296    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:07.568356 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:56:07.568324    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db30dbd_7a9e_41ea_8f4d_681e9576b667.slice/crio-c03699841937c88654ffaee05c2d928a223fc1ef9c74861c032ba8a50765e41d WatchSource:0}: Error finding container c03699841937c88654ffaee05c2d928a223fc1ef9c74861c032ba8a50765e41d: Status 404 returned error can't find the container with id c03699841937c88654ffaee05c2d928a223fc1ef9c74861c032ba8a50765e41d
Nov 03 00:56:08.497121 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.497082    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" event={"ID":"6db30dbd-7a9e-41ea-8f4d-681e9576b667","Type":"ContainerStarted","Data":"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"}
Nov 03 00:56:08.497545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.497195    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" event={"ID":"6db30dbd-7a9e-41ea-8f4d-681e9576b667","Type":"ContainerStarted","Data":"c03699841937c88654ffaee05c2d928a223fc1ef9c74861c032ba8a50765e41d"}
Nov 03 00:56:08.497545 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.497222    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:08.517985 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.517929    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" podStartSLOduration=2.048142032 podStartE2EDuration="2.517916926s" totalImagesPullingTime="469.774894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:08.513733463 +0000 UTC m=+913.727477134" watchObservedRunningTime="2026-11-03 00:56:08.517916926 +0000 UTC m=+913.731660597"
Nov 03 00:56:08.752376 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.752285    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 00:56:08.756334 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.756309    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.775261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.775228    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 00:56:08.836649 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.836582    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.836649 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.836638    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837068 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.836950    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9v5p\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837097    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837277 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837148    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837343 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837322    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837441 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837501 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837489    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.837555 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.837547    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939019 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.938985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939019 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939304 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939080    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9v5p\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939304 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939304 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939128    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939304 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939184    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939304 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939268    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939379    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.939922 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.939897    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.940081 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.940058    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.940218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.940103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.941742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.941718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.942218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.942198    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.947748 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.947724    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:08.947858 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:08.947836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9v5p\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\") pod \"e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") " pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:09.086489 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.086455    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:09.384921 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.384893    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 00:56:09.385596 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:56:09.385572    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31eda088_9b56_4d3f_8ef9_9b9b21f2989e.slice/crio-03e424cc510ef63fbeebf39f15a72263758ce751645caf2406ae4b4f108bd9c4 WatchSource:0}: Error finding container 03e424cc510ef63fbeebf39f15a72263758ce751645caf2406ae4b4f108bd9c4: Status 404 returned error can't find the container with id 03e424cc510ef63fbeebf39f15a72263758ce751645caf2406ae4b4f108bd9c4
Nov 03 00:56:09.388537 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.388504    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:09.388614 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.388568    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:09.388614 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.388597    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:09.537942 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.537887    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" event={"ID":"31eda088-9b56-4d3f-8ef9-9b9b21f2989e","Type":"ContainerStarted","Data":"a46911c5d43a04bd1d33c9b1a146cfe2ea65482868e63527c6d247964e5176ab"}
Nov 03 00:56:09.538373 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.537950    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" event={"ID":"31eda088-9b56-4d3f-8ef9-9b9b21f2989e","Type":"ContainerStarted","Data":"03e424cc510ef63fbeebf39f15a72263758ce751645caf2406ae4b4f108bd9c4"}
Nov 03 00:56:09.563408 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:09.563342    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podStartSLOduration=1.563321629 podStartE2EDuration="1.563321629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:09.557906415 +0000 UTC m=+914.771650099" watchObservedRunningTime="2026-11-03 00:56:09.563321629 +0000 UTC m=+914.777065300"
Nov 03 00:56:09.890183 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:56:09.887867    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:56:10.089186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:10.087139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:10.089186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:10.088935    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:10.089186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:10.089005    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:11.087745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:11.087698    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:11.088316 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:11.087800    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:12.087391 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:12.087358    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:12.087592 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:12.087410    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:13.087833 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:13.087792    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:13.089563 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:13.089525    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:14.087068 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:14.087033    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:14.087272 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:14.087095    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:14.578519 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:14.578478    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:15.087428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:15.087397    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:15.087598 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:15.087448    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:16.087770 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:16.087728    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:56:16.088191 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:16.087801    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:56:18.128476 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:18.128304    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:56:18.128476 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:18.128380    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:56:18.174665 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:18.174628    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:18.175003 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:18.174978    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:18.175758 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:18.175740    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 00:56:31.528753 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.528710    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:56:31.574226 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.565543    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.576809 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.565441    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:56:31.578369 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.578348    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-65e5ec-serving-cert\""
Nov 03 00:56:31.600791 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.600763    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.600940 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.600803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.601022 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.600999    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.601075 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.601059    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzh5f\" (UniqueName: \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.704130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.704079    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.704367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.704191    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzh5f\" (UniqueName: \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.704367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.704339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.704488 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.704382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.705362 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.705329    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.708618 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.708586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.714477 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.714418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.717461 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.717419    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzh5f\" (UniqueName: \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\") pod \"maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:31.896029 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:31.896001    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:32.188691 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.188660    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:56:32.190585 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:56:32.190544    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eef6aad_ff88_4d63_adcd_354c66d77203.slice/crio-8037dacba363c77cb5c00f65d7d729e5337048042714d90159c119649454cfdc WatchSource:0}: Error finding container 8037dacba363c77cb5c00f65d7d729e5337048042714d90159c119649454cfdc: Status 404 returned error can't find the container with id 8037dacba363c77cb5c00f65d7d729e5337048042714d90159c119649454cfdc
Nov 03 00:56:32.192571 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.192547    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:56:32.773218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.773180    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" event={"ID":"4eef6aad-ff88-4d63-adcd-354c66d77203","Type":"ContainerStarted","Data":"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"}
Nov 03 00:56:32.773629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.773249    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:32.773629 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.773267    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" event={"ID":"4eef6aad-ff88-4d63-adcd-354c66d77203","Type":"ContainerStarted","Data":"8037dacba363c77cb5c00f65d7d729e5337048042714d90159c119649454cfdc"}
Nov 03 00:56:32.796574 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:32.796520    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" podStartSLOduration=1.363027093 podStartE2EDuration="1.796505471s" totalImagesPullingTime="433.478378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:32.792374484 +0000 UTC m=+938.006118156" watchObservedRunningTime="2026-11-03 00:56:32.796505471 +0000 UTC m=+938.010249146"
Nov 03 00:56:38.840130 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:38.840051    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:56:39.888929 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:56:39.888892    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:56:54.838796 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.838757    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 00:56:54.843239 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.843216    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.851744 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.851293    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:56:54.855127 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.855105    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:56:54.855286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.855221    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9vdqv\""
Nov 03 00:56:54.855286 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.855276    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:56:54.857826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.857802    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 00:56:54.991284 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991464 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991407    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991463    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8dd\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991685 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991666    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991738 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991728    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991796 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:54.991948 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:54.991925    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.092870 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.092798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.092874    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.092912    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8dd\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093011 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.092985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.093018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.093045    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093248 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.093066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.093782 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.093756    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.095514 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.095480    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:56:55.095620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.095524    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:56:55.095620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.095541    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:56:55.095857 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.095836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.101124 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.101101    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.101714 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.101696    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8dd\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.105106 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.105081    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.105331 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.105310    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.105386 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.105314    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mktb8\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.163732 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.163698    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9vdqv\""
Nov 03 00:56:55.171263 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.171236    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:56:55.503646 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:55.503613    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 00:56:55.505040 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:56:55.505014    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083bc720_a6ca_480f_ba2d_14f88600f1f4.slice/crio-6bca525c1c8e49758db6b1a63721dbe8cd2326e0250b4d9969bc27c9325498f6 WatchSource:0}: Error finding container 6bca525c1c8e49758db6b1a63721dbe8cd2326e0250b4d9969bc27c9325498f6: Status 404 returned error can't find the container with id 6bca525c1c8e49758db6b1a63721dbe8cd2326e0250b4d9969bc27c9325498f6
Nov 03 00:56:56.050999 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:56.050955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" event={"ID":"083bc720-a6ca-480f-ba2d-14f88600f1f4","Type":"ContainerStarted","Data":"6bca525c1c8e49758db6b1a63721dbe8cd2326e0250b4d9969bc27c9325498f6"}
Nov 03 00:56:58.372901 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.372867    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:58.373760 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.373718    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" podUID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" containerName="maas-api" containerID="cri-o://fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0" gracePeriod=30
Nov 03 00:56:58.854826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.852686    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:58.893265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.891229    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\") pod \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") "
Nov 03 00:56:58.893265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.891308    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\") pod \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") "
Nov 03 00:56:58.893265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.891363    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\" (UniqueName: \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\") pod \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") "
Nov 03 00:56:58.893265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.891522    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\") pod \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\" (UID: \"6db30dbd-7a9e-41ea-8f4d-681e9576b667\") "
Nov 03 00:56:58.893265 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.892819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca" pod "6db30dbd-7a9e-41ea-8f4d-681e9576b667" (UID: "6db30dbd-7a9e-41ea-8f4d-681e9576b667"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:56:58.899254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.899221    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs" pod "6db30dbd-7a9e-41ea-8f4d-681e9576b667" (UID: "6db30dbd-7a9e-41ea-8f4d-681e9576b667"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:58.899790 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.899766    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls" pod "6db30dbd-7a9e-41ea-8f4d-681e9576b667" (UID: "6db30dbd-7a9e-41ea-8f4d-681e9576b667"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:58.900656 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:58.900632    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr" pod "6db30dbd-7a9e-41ea-8f4d-681e9576b667" (UID: "6db30dbd-7a9e-41ea-8f4d-681e9576b667"). InnerVolumeSpecName "kube-api-access-46jrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:56:59.007704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.001764    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:56:59.007704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.001858    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6db30dbd-7a9e-41ea-8f4d-681e9576b667-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:56:59.007704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.001901    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46jrr\" (UniqueName: \"kubernetes.io/projected/6db30dbd-7a9e-41ea-8f4d-681e9576b667-kube-api-access-46jrr\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:56:59.007704 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.001916    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6db30dbd-7a9e-41ea-8f4d-681e9576b667-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:56:59.084532 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.084498    4531 generic.go:356] "Generic (PLEG): container finished" podID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" containerID="fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0" exitCode=0
Nov 03 00:56:59.084745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.084553    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"
Nov 03 00:56:59.084745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.084581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" event={"ID":"6db30dbd-7a9e-41ea-8f4d-681e9576b667","Type":"ContainerDied","Data":"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"}
Nov 03 00:56:59.084745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.084641    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4" event={"ID":"6db30dbd-7a9e-41ea-8f4d-681e9576b667","Type":"ContainerDied","Data":"c03699841937c88654ffaee05c2d928a223fc1ef9c74861c032ba8a50765e41d"}
Nov 03 00:56:59.084745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.084672    4531 scope.go:122] "RemoveContainer" containerID="fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"
Nov 03 00:56:59.098188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.098149    4531 scope.go:122] "RemoveContainer" containerID="fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"
Nov 03 00:56:59.098520 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:56:59.098483    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0\": container with ID starting with fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0 not found: ID does not exist" containerID="fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"
Nov 03 00:56:59.099062 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.099029    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0"} err="failed to get container status \"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0\": rpc error: code = NotFound desc = could not find container \"fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0\": container with ID starting with fbcf29e13af44b8475074afa8a1e94c544295f70f0234670878bd62f64001ec0 not found: ID does not exist"
Nov 03 00:56:59.106884 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.106856    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:59.109393 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.109374    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51119c-7d7689f997-pw2g4"]
Nov 03 00:56:59.247097 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.247065    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:59.247209 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:56:59.247131    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:00.154182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.154130    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" event={"ID":"083bc720-a6ca-480f-ba2d-14f88600f1f4","Type":"ContainerStarted","Data":"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"}
Nov 03 00:57:00.159211 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.155136    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:57:00.159211 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.156465    4531 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mktb8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:57:00.159211 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.156523    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:57:00.180598 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.180546    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" podStartSLOduration=2.441796318 podStartE2EDuration="6.180532358s" totalImagesPullingTime="3.73873604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:00.177304265 +0000 UTC m=+965.391047941" watchObservedRunningTime="2026-11-03 00:57:00.180532358 +0000 UTC m=+965.394276030"
Nov 03 00:57:00.904577 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:00.904544    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" path="/var/lib/kubelet/pods/6db30dbd-7a9e-41ea-8f4d-681e9576b667/volumes"
Nov 03 00:57:01.214065 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:01.213976    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 00:57:09.881383 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:57:09.881345    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:57:10.383762 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:10.383723    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:57:10.384337 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:10.384271    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" podUID="4eef6aad-ff88-4d63-adcd-354c66d77203" containerName="maas-api" containerID="cri-o://1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b" gracePeriod=30
Nov 03 00:57:10.916367 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:10.916334    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:57:11.025146 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025110    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\") pod \"4eef6aad-ff88-4d63-adcd-354c66d77203\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") "
Nov 03 00:57:11.025345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025190    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\" (UniqueName: \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\") pod \"4eef6aad-ff88-4d63-adcd-354c66d77203\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") "
Nov 03 00:57:11.025345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025257    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\") pod \"4eef6aad-ff88-4d63-adcd-354c66d77203\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") "
Nov 03 00:57:11.025345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025320    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\") pod \"4eef6aad-ff88-4d63-adcd-354c66d77203\" (UID: \"4eef6aad-ff88-4d63-adcd-354c66d77203\") "
Nov 03 00:57:11.025641 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025610    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca" pod "4eef6aad-ff88-4d63-adcd-354c66d77203" (UID: "4eef6aad-ff88-4d63-adcd-354c66d77203"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:11.025977 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.025956    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eef6aad-ff88-4d63-adcd-354c66d77203-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:11.027400 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.027368    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs" pod "4eef6aad-ff88-4d63-adcd-354c66d77203" (UID: "4eef6aad-ff88-4d63-adcd-354c66d77203"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:11.027646 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.027619    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls" pod "4eef6aad-ff88-4d63-adcd-354c66d77203" (UID: "4eef6aad-ff88-4d63-adcd-354c66d77203"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:11.027810 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.027790    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f" pod "4eef6aad-ff88-4d63-adcd-354c66d77203" (UID: "4eef6aad-ff88-4d63-adcd-354c66d77203"). InnerVolumeSpecName "kube-api-access-mzh5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:11.132212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.130300    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:11.132212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.130342    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzh5f\" (UniqueName: \"kubernetes.io/projected/4eef6aad-ff88-4d63-adcd-354c66d77203-kube-api-access-mzh5f\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:11.132212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.130359    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef6aad-ff88-4d63-adcd-354c66d77203-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:11.290006 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.289914    4531 generic.go:356] "Generic (PLEG): container finished" podID="4eef6aad-ff88-4d63-adcd-354c66d77203" containerID="1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b" exitCode=0
Nov 03 00:57:11.290006 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.289958    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" event={"ID":"4eef6aad-ff88-4d63-adcd-354c66d77203","Type":"ContainerDied","Data":"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"}
Nov 03 00:57:11.290006 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.290004    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf" event={"ID":"4eef6aad-ff88-4d63-adcd-354c66d77203","Type":"ContainerDied","Data":"8037dacba363c77cb5c00f65d7d729e5337048042714d90159c119649454cfdc"}
Nov 03 00:57:11.290328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.290030    4531 scope.go:122] "RemoveContainer" containerID="1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"
Nov 03 00:57:11.290328 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.289959    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"
Nov 03 00:57:11.304411 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.304388    4531 scope.go:122] "RemoveContainer" containerID="1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"
Nov 03 00:57:11.304983 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:57:11.304954    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b\": container with ID starting with 1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b not found: ID does not exist" containerID="1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"
Nov 03 00:57:11.305071 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.304994    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b"} err="failed to get container status \"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b\": rpc error: code = NotFound desc = could not find container \"1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b\": container with ID starting with 1320b52986e667d7b6232347136aa70d9ca8179087163be63ab3038b762cd04b not found: ID does not exist"
Nov 03 00:57:11.315467 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.315443    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:57:11.325846 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:11.325822    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65e5ec-5f8c9964c7-rnftf"]
Nov 03 00:57:12.904144 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:12.904109    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eef6aad-ff88-4d63-adcd-354c66d77203" path="/var/lib/kubelet/pods/4eef6aad-ff88-4d63-adcd-354c66d77203/volumes"
Nov 03 00:57:33.542392 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:33.542354    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:57:33.549880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:33.542615    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" podUID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" containerName="maas-api" containerID="cri-o://5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb" gracePeriod=30
Nov 03 00:57:34.076396 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.076366    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:57:34.253546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.253451    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\" (UniqueName: \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\") pod \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") "
Nov 03 00:57:34.253546 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.253518    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\") pod \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") "
Nov 03 00:57:34.253789 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.253590    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\") pod \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") "
Nov 03 00:57:34.253789 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.253660    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\") pod \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\" (UID: \"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6\") "
Nov 03 00:57:34.254799 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.254453    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca" pod "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" (UID: "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:34.268188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.266030    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls" pod "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" (UID: "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:34.268188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.267698    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf" pod "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" (UID: "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6"). InnerVolumeSpecName "kube-api-access-8xwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:34.277476 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.272817    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs" pod "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" (UID: "d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:34.355620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.355582    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwkf\" (UniqueName: \"kubernetes.io/projected/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-kube-api-access-8xwkf\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:34.355620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.355614    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:34.355620 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.355629    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:34.355934 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.355643    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:57:34.547179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.547066    4531 generic.go:356] "Generic (PLEG): container finished" podID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" containerID="5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb" exitCode=0
Nov 03 00:57:34.547179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.547112    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"
Nov 03 00:57:34.547179 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.547131    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" event={"ID":"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6","Type":"ContainerDied","Data":"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"}
Nov 03 00:57:34.547759 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.547219    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x" event={"ID":"d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6","Type":"ContainerDied","Data":"48851326c86bd89c530f1ab82c4ec5178c721cffd1c0a8dabec8a1e0fcc04b61"}
Nov 03 00:57:34.547759 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.547245    4531 scope.go:122] "RemoveContainer" containerID="5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"
Nov 03 00:57:34.563582 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.563430    4531 scope.go:122] "RemoveContainer" containerID="5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"
Nov 03 00:57:34.563810 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:57:34.563787    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb\": container with ID starting with 5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb not found: ID does not exist" containerID="5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"
Nov 03 00:57:34.563879 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.563823    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb"} err="failed to get container status \"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb\": rpc error: code = NotFound desc = could not find container \"5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb\": container with ID starting with 5af265eb836a600c894b96fd8b33d61213a85eed90a6106ebe9217f225d1e6bb not found: ID does not exist"
Nov 03 00:57:34.574876 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.574848    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:57:34.580861 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.580821    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-q4d5x"]
Nov 03 00:57:34.907965 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:34.907918    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" path="/var/lib/kubelet/pods/d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6/volumes"
Nov 03 00:57:39.880778 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:57:39.880739    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:57:42.184204 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184100    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184743    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4eef6aad-ff88-4d63-adcd-354c66d77203" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184756    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eef6aad-ff88-4d63-adcd-354c66d77203" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184772    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184777    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184811    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184820    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184899    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db30dbd-7a9e-41ea-8f4d-681e9576b667" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184909    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eef6aad-ff88-4d63-adcd-354c66d77203" containerName="maas-api"
Nov 03 00:57:42.185434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.184917    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95fa4a0-0d51-4d4a-9954-b4e08a4df4f6" containerName="maas-api"
Nov 03 00:57:42.188515 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.188491    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:57:42.188656 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.188634    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.194314 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.194233    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a2d8e5-serving-cert\""
Nov 03 00:57:42.311875 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.311834    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.312085 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.311946    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.312085 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.311998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.312246 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.312230    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6v4c\" (UniqueName: \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.415467 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.415428    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6v4c\" (UniqueName: \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.415961 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.415925    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.416085 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.416001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.416085 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.416046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.418207 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.418127    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.434826 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.434744    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.435807 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.435777    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.440816 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.440787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6v4c\" (UniqueName: \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\") pod \"maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.507525 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.507436    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:42.802747 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:42.802715    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:57:42.803051 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:57:42.803030    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad51691_8016_4453_aefe_531774ccb0f8.slice/crio-02aaff0a3d4577a26cd60bf419e2e9c2660a872f1ae20b8c86e2957df9c794fe WatchSource:0}: Error finding container 02aaff0a3d4577a26cd60bf419e2e9c2660a872f1ae20b8c86e2957df9c794fe: Status 404 returned error can't find the container with id 02aaff0a3d4577a26cd60bf419e2e9c2660a872f1ae20b8c86e2957df9c794fe
Nov 03 00:57:43.645182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:43.644440    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" event={"ID":"6ad51691-8016-4453-aefe-531774ccb0f8","Type":"ContainerStarted","Data":"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"}
Nov 03 00:57:43.645182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:43.644507    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" event={"ID":"6ad51691-8016-4453-aefe-531774ccb0f8","Type":"ContainerStarted","Data":"02aaff0a3d4577a26cd60bf419e2e9c2660a872f1ae20b8c86e2957df9c794fe"}
Nov 03 00:57:43.645182 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:43.644554    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:43.692884 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:43.692450    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" podStartSLOduration=1.20258354 podStartE2EDuration="1.692435638s" totalImagesPullingTime="489.852098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:43.690584029 +0000 UTC m=+1008.904327702" watchObservedRunningTime="2026-11-03 00:57:43.692435638 +0000 UTC m=+1008.906179311"
Nov 03 00:57:48.316688 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.316654    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:57:48.322042 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.322020    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.333506 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.333476    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:57:48.334250 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.334191    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd03e395-serving-cert\""
Nov 03 00:57:48.453055 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.453013    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.453296 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.453222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.453419 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.453404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.453534 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.453518    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfkx\" (UniqueName: \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.554468 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.554421    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.554746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.554542    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfkx\" (UniqueName: \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.554746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.554595    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.554746 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.554707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.555893 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.555869    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.558373 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.558344    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.558558 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.558535    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.564552 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.564526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfkx\" (UniqueName: \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\") pod \"maas-api-e2e-mt-dd03e395-55957764d5-sfk2r\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:48.654152 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:48.654117    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:49.010215 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:49.008386    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:57:49.666292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:49.666259    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:57:49.713461 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:49.713374    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" event={"ID":"3fa8ef30-2e9b-4805-9ecd-edf5d560306c","Type":"ContainerStarted","Data":"a3a71d163229ab66b996db3a334359c57743300f1088c49e4e0fd4696bdf0c29"}
Nov 03 00:57:49.713461 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:49.713424    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" event={"ID":"3fa8ef30-2e9b-4805-9ecd-edf5d560306c","Type":"ContainerStarted","Data":"9063c8dd9c9f2504d662565243a746417e84cdf60a77eb568048482289f0b38c"}
Nov 03 00:57:49.714137 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:49.714117    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:56.733227 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:56.733197    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:57:56.752528 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:57:56.752471    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" podStartSLOduration=8.291446976 podStartE2EDuration="8.752455534s" totalImagesPullingTime="461.008558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:49.747427383 +0000 UTC m=+1014.961171055" watchObservedRunningTime="2026-11-03 00:57:56.752455534 +0000 UTC m=+1021.966199206"
Nov 03 00:58:10.044585 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:58:10.044396    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:58:21.509497 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.509463    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:58:21.515260 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.515237    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.532425 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.532389    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.532603 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.532495    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjrs\" (UniqueName: \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.532681 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.532634    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.532741 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.532681    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.533386 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.533363    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-eed369-serving-cert\""
Nov 03 00:58:21.559769 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.559729    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:58:21.635353 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.635298    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.635681 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.635639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.643354 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.643322    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.643633 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.643595    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.643955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.643936    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjrs\" (UniqueName: \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.646229 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:58:21.646208    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-serving-cert: secret "maas-api-e2e-api-b-eed369-serving-cert" not found
Nov 03 00:58:21.646740 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:58:21.646726    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls podName:0dead7dc-86ba-43fd-a229-ec54ded2769c nodeName:}" failed. No retries permitted until 2026-11-03 00:58:22.146703035 +0000 UTC m=+1047.360446698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls") pod "maas-api-e2e-api-b-eed369-799649b69b-fncb5" (UID: "0dead7dc-86ba-43fd-a229-ec54ded2769c") : secret "maas-api-e2e-api-b-eed369-serving-cert" not found
Nov 03 00:58:21.658929 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.658860    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:21.659189 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:21.659085    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjrs\" (UniqueName: \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:22.153184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:22.152199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:22.203261 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:22.200391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") pod \"maas-api-e2e-api-b-eed369-799649b69b-fncb5\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:22.448037 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:22.447944    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:22.733818 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:58:22.733786    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dead7dc_86ba_43fd_a229_ec54ded2769c.slice/crio-151ebc09cf3f9f82db4533166419f228c3664feb3627afe62f273785f349823c WatchSource:0}: Error finding container 151ebc09cf3f9f82db4533166419f228c3664feb3627afe62f273785f349823c: Status 404 returned error can't find the container with id 151ebc09cf3f9f82db4533166419f228c3664feb3627afe62f273785f349823c
Nov 03 00:58:22.734589 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:22.734499    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:58:23.242707 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.242662    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" event={"ID":"0dead7dc-86ba-43fd-a229-ec54ded2769c","Type":"ContainerStarted","Data":"151ebc09cf3f9f82db4533166419f228c3664feb3627afe62f273785f349823c"}
Nov 03 00:58:23.814724 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.814692    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 00:58:23.819212 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.819148    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.840853 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.838538    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-07f5f6-serving-cert\""
Nov 03 00:58:23.847479 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.847451    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 00:58:23.879601 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.879553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.879784 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.879610    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxnb\" (UniqueName: \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.879784 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.879748    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.879918 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.879785    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.981111 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.981076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.981309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.981126    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxnb\" (UniqueName: \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.981309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.981245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.981309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.981277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.981959 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.981938    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.983660 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.983637    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.983779 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.983747    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:23.990191 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:23.990152    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxnb\" (UniqueName: \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\") pod \"maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:24.148928 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:24.148888    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:24.310837 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:24.310797    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" event={"ID":"0dead7dc-86ba-43fd-a229-ec54ded2769c","Type":"ContainerStarted","Data":"51ab7900730cf5da0d21bc6624316b230cada0b35ea15eafb6282728025c82d7"}
Nov 03 00:58:24.311178 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:24.311139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:24.335748 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:24.335688    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" podStartSLOduration=2.7808606129999998 podStartE2EDuration="3.335669502s" totalImagesPullingTime="554.808889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:24.333460503 +0000 UTC m=+1049.547204177" watchObservedRunningTime="2026-11-03 00:58:24.335669502 +0000 UTC m=+1049.549413176"
Nov 03 00:58:24.444235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:24.444138    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 00:58:25.319345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:25.319258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" event={"ID":"2334dcb7-06f3-4801-ac01-fff02d1e4fea","Type":"ContainerStarted","Data":"ba5b4a4eafa03ca81721790d599fae587a024fac8a757f29f707c9a87db49888"}
Nov 03 00:58:25.319345 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:25.319313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" event={"ID":"2334dcb7-06f3-4801-ac01-fff02d1e4fea","Type":"ContainerStarted","Data":"f80af7b0dc578ac83fa1055027420d745ff11c7f67d0f92fc8e8a02c451ee303"}
Nov 03 00:58:25.343140 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:25.343063    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" podStartSLOduration=1.915046431 podStartE2EDuration="2.343045624s" totalImagesPullingTime="427.999193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:25.338114795 +0000 UTC m=+1050.551858468" watchObservedRunningTime="2026-11-03 00:58:25.343045624 +0000 UTC m=+1050.556789297"
Nov 03 00:58:26.327235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:26.327202    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:26.499256 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:26.498745    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:58:26.499256 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:26.498987    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" podUID="6ad51691-8016-4453-aefe-531774ccb0f8" containerName="maas-api" containerID="cri-o://9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1" gracePeriod=30
Nov 03 00:58:27.055089 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.055056    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:58:27.127551 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.127515    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\") pod \"6ad51691-8016-4453-aefe-531774ccb0f8\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") "
Nov 03 00:58:27.128205 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.128179    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\") pod \"6ad51691-8016-4453-aefe-531774ccb0f8\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") "
Nov 03 00:58:27.128788 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.128769    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\") pod \"6ad51691-8016-4453-aefe-531774ccb0f8\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") "
Nov 03 00:58:27.129410 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.129389    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\" (UniqueName: \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\") pod \"6ad51691-8016-4453-aefe-531774ccb0f8\" (UID: \"6ad51691-8016-4453-aefe-531774ccb0f8\") "
Nov 03 00:58:27.130570 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.128673    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca" pod "6ad51691-8016-4453-aefe-531774ccb0f8" (UID: "6ad51691-8016-4453-aefe-531774ccb0f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:58:27.147847 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.147816    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs" pod "6ad51691-8016-4453-aefe-531774ccb0f8" (UID: "6ad51691-8016-4453-aefe-531774ccb0f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:58:27.148115 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.148091    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls" pod "6ad51691-8016-4453-aefe-531774ccb0f8" (UID: "6ad51691-8016-4453-aefe-531774ccb0f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:58:27.149339 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.149312    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c" pod "6ad51691-8016-4453-aefe-531774ccb0f8" (UID: "6ad51691-8016-4453-aefe-531774ccb0f8"). InnerVolumeSpecName "kube-api-access-z6v4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:27.231717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.231672    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6v4c\" (UniqueName: \"kubernetes.io/projected/6ad51691-8016-4453-aefe-531774ccb0f8-kube-api-access-z6v4c\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:27.231717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.231705    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:27.231717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.231718    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad51691-8016-4453-aefe-531774ccb0f8-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:27.231717 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.231727    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad51691-8016-4453-aefe-531774ccb0f8-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:27.338221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.338187    4531 generic.go:356] "Generic (PLEG): container finished" podID="6ad51691-8016-4453-aefe-531774ccb0f8" containerID="9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1" exitCode=0
Nov 03 00:58:27.338725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.338241    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"
Nov 03 00:58:27.338725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.338248    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" event={"ID":"6ad51691-8016-4453-aefe-531774ccb0f8","Type":"ContainerDied","Data":"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"}
Nov 03 00:58:27.338725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.338327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq" event={"ID":"6ad51691-8016-4453-aefe-531774ccb0f8","Type":"ContainerDied","Data":"02aaff0a3d4577a26cd60bf419e2e9c2660a872f1ae20b8c86e2957df9c794fe"}
Nov 03 00:58:27.338725 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.338349    4531 scope.go:122] "RemoveContainer" containerID="9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"
Nov 03 00:58:27.354626 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.354607    4531 scope.go:122] "RemoveContainer" containerID="9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"
Nov 03 00:58:27.355004 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:58:27.354982    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1\": container with ID starting with 9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1 not found: ID does not exist" containerID="9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"
Nov 03 00:58:27.355070 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.355013    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1"} err="failed to get container status \"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1\": rpc error: code = NotFound desc = could not find container \"9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1\": container with ID starting with 9beaebd9ad0899ebc75d68ea759fc6b633afa77f3af25183bdfb271e86b2eed1 not found: ID does not exist"
Nov 03 00:58:27.369947 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.369916    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:58:27.375428 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:27.375406    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a2d8e5-6f6d9ff775-hqxxq"]
Nov 03 00:58:28.927184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:28.927060    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad51691-8016-4453-aefe-531774ccb0f8" path="/var/lib/kubelet/pods/6ad51691-8016-4453-aefe-531774ccb0f8/volumes"
Nov 03 00:58:30.338880 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:30.338849    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:58:32.363959 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:32.363930    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 00:58:39.879243 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:58:39.879199    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:58:56.649257 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:56.649217    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:58:56.649777 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:56.649505    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerName="maas-api" containerID="cri-o://a3a71d163229ab66b996db3a334359c57743300f1088c49e4e0fd4696bdf0c29" gracePeriod=30
Nov 03 00:58:56.728057 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:56.727715    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": dial tcp 10.133.0.84:8443: connect: connection refused"
Nov 03 00:58:56.822232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:56.821946    4531 generic.go:356] "Generic (PLEG): container finished" podID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerID="a3a71d163229ab66b996db3a334359c57743300f1088c49e4e0fd4696bdf0c29" exitCode=0
Nov 03 00:58:56.822232 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:56.822119    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" event={"ID":"3fa8ef30-2e9b-4805-9ecd-edf5d560306c","Type":"ContainerDied","Data":"a3a71d163229ab66b996db3a334359c57743300f1088c49e4e0fd4696bdf0c29"}
Nov 03 00:58:57.150504 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.150089    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:58:57.231513 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.231462    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\") pod \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") "
Nov 03 00:58:57.231823 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.231799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\") pod \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") "
Nov 03 00:58:57.231918 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.231899    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\" (UniqueName: \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\") pod \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") "
Nov 03 00:58:57.232014 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.231998    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\") pod \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\" (UID: \"3fa8ef30-2e9b-4805-9ecd-edf5d560306c\") "
Nov 03 00:58:57.233235 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.233206    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca" pod "3fa8ef30-2e9b-4805-9ecd-edf5d560306c" (UID: "3fa8ef30-2e9b-4805-9ecd-edf5d560306c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:58:57.238858 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.237921    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx" pod "3fa8ef30-2e9b-4805-9ecd-edf5d560306c" (UID: "3fa8ef30-2e9b-4805-9ecd-edf5d560306c"). InnerVolumeSpecName "kube-api-access-rcfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:57.240035 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.240008    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs" pod "3fa8ef30-2e9b-4805-9ecd-edf5d560306c" (UID: "3fa8ef30-2e9b-4805-9ecd-edf5d560306c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:58:57.244258 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.244209    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls" pod "3fa8ef30-2e9b-4805-9ecd-edf5d560306c" (UID: "3fa8ef30-2e9b-4805-9ecd-edf5d560306c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:58:57.340627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.339474    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:57.340627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.340547    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfkx\" (UniqueName: \"kubernetes.io/projected/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-kube-api-access-rcfkx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:57.340627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.340573    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:57.340627 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.340589    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa8ef30-2e9b-4805-9ecd-edf5d560306c-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:58:57.831029 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.830998    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r" event={"ID":"3fa8ef30-2e9b-4805-9ecd-edf5d560306c","Type":"ContainerDied","Data":"9063c8dd9c9f2504d662565243a746417e84cdf60a77eb568048482289f0b38c"}
Nov 03 00:58:57.831503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.831064    4531 scope.go:122] "RemoveContainer" containerID="a3a71d163229ab66b996db3a334359c57743300f1088c49e4e0fd4696bdf0c29"
Nov 03 00:58:57.831503 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.831008    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"
Nov 03 00:58:57.867770 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.867736    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:58:57.872955 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:57.872911    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd03e395-55957764d5-sfk2r"]
Nov 03 00:58:58.906636 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:58:58.906602    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" path="/var/lib/kubelet/pods/3fa8ef30-2e9b-4805-9ecd-edf5d560306c/volumes"
Nov 03 00:59:09.970206 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:59:09.966702    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:59:16.784231 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:16.784183    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:59:16.784699 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:16.784491    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" podUID="0dead7dc-86ba-43fd-a229-ec54ded2769c" containerName="maas-api" containerID="cri-o://51ab7900730cf5da0d21bc6624316b230cada0b35ea15eafb6282728025c82d7" gracePeriod=30
Nov 03 00:59:17.074282 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.073106    4531 generic.go:356] "Generic (PLEG): container finished" podID="0dead7dc-86ba-43fd-a229-ec54ded2769c" containerID="51ab7900730cf5da0d21bc6624316b230cada0b35ea15eafb6282728025c82d7" exitCode=0
Nov 03 00:59:17.074282 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.073200    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" event={"ID":"0dead7dc-86ba-43fd-a229-ec54ded2769c","Type":"ContainerDied","Data":"51ab7900730cf5da0d21bc6624316b230cada0b35ea15eafb6282728025c82d7"}
Nov 03 00:59:17.282377 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.282349    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:59:17.429618 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.429525    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\") pod \"0dead7dc-86ba-43fd-a229-ec54ded2769c\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") "
Nov 03 00:59:17.429618 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.429613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\") pod \"0dead7dc-86ba-43fd-a229-ec54ded2769c\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") "
Nov 03 00:59:17.429866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.429757    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") pod \"0dead7dc-86ba-43fd-a229-ec54ded2769c\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") "
Nov 03 00:59:17.429866 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.429855    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\" (UniqueName: \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\") pod \"0dead7dc-86ba-43fd-a229-ec54ded2769c\" (UID: \"0dead7dc-86ba-43fd-a229-ec54ded2769c\") "
Nov 03 00:59:17.430049 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.430023    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca" pod "0dead7dc-86ba-43fd-a229-ec54ded2769c" (UID: "0dead7dc-86ba-43fd-a229-ec54ded2769c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:59:17.430532 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.430507    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dead7dc-86ba-43fd-a229-ec54ded2769c-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:59:17.431982 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.431951    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls" pod "0dead7dc-86ba-43fd-a229-ec54ded2769c" (UID: "0dead7dc-86ba-43fd-a229-ec54ded2769c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:59:17.432074 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.431951    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs" pod "0dead7dc-86ba-43fd-a229-ec54ded2769c" (UID: "0dead7dc-86ba-43fd-a229-ec54ded2769c"). InnerVolumeSpecName "kube-api-access-shjrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:59:17.432119 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.432070    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs" pod "0dead7dc-86ba-43fd-a229-ec54ded2769c" (UID: "0dead7dc-86ba-43fd-a229-ec54ded2769c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:59:17.532939 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.532901    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:59:17.532939 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.532938    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjrs\" (UniqueName: \"kubernetes.io/projected/0dead7dc-86ba-43fd-a229-ec54ded2769c-kube-api-access-shjrs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:59:17.532939 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:17.532954    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dead7dc-86ba-43fd-a229-ec54ded2769c-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 00:59:18.082195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.082150    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5" event={"ID":"0dead7dc-86ba-43fd-a229-ec54ded2769c","Type":"ContainerDied","Data":"151ebc09cf3f9f82db4533166419f228c3664feb3627afe62f273785f349823c"}
Nov 03 00:59:18.082195 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.082154    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"
Nov 03 00:59:18.082668 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.082226    4531 scope.go:122] "RemoveContainer" containerID="51ab7900730cf5da0d21bc6624316b230cada0b35ea15eafb6282728025c82d7"
Nov 03 00:59:18.110663 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.110637    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:59:18.117745 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.117718    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-eed369-799649b69b-fncb5"]
Nov 03 00:59:18.928593 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:18.928085    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dead7dc-86ba-43fd-a229-ec54ded2769c" path="/var/lib/kubelet/pods/0dead7dc-86ba-43fd-a229-ec54ded2769c/volumes"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.754732    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755604    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerName="maas-api"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755619    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerName="maas-api"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755660    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0dead7dc-86ba-43fd-a229-ec54ded2769c" containerName="maas-api"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755667    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dead7dc-86ba-43fd-a229-ec54ded2769c" containerName="maas-api"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755685    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6ad51691-8016-4453-aefe-531774ccb0f8" containerName="maas-api"
Nov 03 00:59:21.755720 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755692    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad51691-8016-4453-aefe-531774ccb0f8" containerName="maas-api"
Nov 03 00:59:21.756575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755827    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad51691-8016-4453-aefe-531774ccb0f8" containerName="maas-api"
Nov 03 00:59:21.756575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755846    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa8ef30-2e9b-4805-9ecd-edf5d560306c" containerName="maas-api"
Nov 03 00:59:21.756575 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.755858    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dead7dc-86ba-43fd-a229-ec54ded2769c" containerName="maas-api"
Nov 03 00:59:21.803043 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.802528    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 00:59:21.803829 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.802934    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:21.836221 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.835050    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7406a9-serving-cert\""
Nov 03 00:59:21.908099 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.908062    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:21.908317 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.908139    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:21.908317 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.908182    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvwv\" (UniqueName: \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:21.908434 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:21.908363    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.013243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.012123    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.013243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.012236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.013243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.012325    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.013243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.012352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvwv\" (UniqueName: \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.018557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.013648    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.053186 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.040754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.072200 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.063725    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.073058 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.072992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvwv\" (UniqueName: \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\") pod \"maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.133188 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.131411    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:22.524198 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:22.524136    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 00:59:22.525302 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:59:22.525275    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5dcbf9_3eab_4d3e_8b1e_fbddc9ce3a2f.slice/crio-e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f WatchSource:0}: Error finding container e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f: Status 404 returned error can't find the container with id e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f
Nov 03 00:59:23.192983 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:23.192946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" event={"ID":"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f","Type":"ContainerStarted","Data":"d732ca02e131741a7b44c99779037045ceb6ec6ac56622e0e763df1a87c85945"}
Nov 03 00:59:23.193500 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:23.193005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" event={"ID":"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f","Type":"ContainerStarted","Data":"e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f"}
Nov 03 00:59:23.194056 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:23.193923    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:23.215591 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:23.215539    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" podStartSLOduration=1.765437074 podStartE2EDuration="2.215525592s" totalImagesPullingTime="450.088518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:23.211349669 +0000 UTC m=+1108.425093343" watchObservedRunningTime="2026-11-03 00:59:23.215525592 +0000 UTC m=+1108.429269263"
Nov 03 00:59:24.972500 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:24.972466    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 00:59:24.997520 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:24.997302    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 00:59:24.997891 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:24.997871    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.010765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.009832    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmttc\" (UniqueName: \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.010765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.009925    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.010765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.010015    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.010765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.010044    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.019266 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.019243    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-47369d-serving-cert\""
Nov 03 00:59:25.111529 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.111495    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.111676 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.111542    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.111766 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.111751    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmttc\" (UniqueName: \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.111875 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.111859    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.112249 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.112216    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.114081 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.114055    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.114408 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.114379    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.140419 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.140389    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmttc\" (UniqueName: \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\") pod \"maas-api-e2e-api-b-47369d-544b548f99-2hzjz\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.317380 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.317271    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:25.613337 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:25.613312    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 00:59:25.614871 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:59:25.614841    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd336b05a_dab8_4523_8e64_25d71ff87726.slice/crio-371341a812fb4d081941452a51f2fb07123721101c40e88827395a458b88c549 WatchSource:0}: Error finding container 371341a812fb4d081941452a51f2fb07123721101c40e88827395a458b88c549: Status 404 returned error can't find the container with id 371341a812fb4d081941452a51f2fb07123721101c40e88827395a458b88c549
Nov 03 00:59:26.218765 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:26.218734    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" event={"ID":"d336b05a-dab8-4523-8e64-25d71ff87726","Type":"ContainerStarted","Data":"371341a812fb4d081941452a51f2fb07123721101c40e88827395a458b88c549"}
Nov 03 00:59:27.231662 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:27.231626    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" event={"ID":"d336b05a-dab8-4523-8e64-25d71ff87726","Type":"ContainerStarted","Data":"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"}
Nov 03 00:59:27.232220 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:27.231777    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:27.256744 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:27.256687    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" podStartSLOduration=2.712821834 podStartE2EDuration="3.25667366s" totalImagesPullingTime="543.851826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:27.252477209 +0000 UTC m=+1112.466220895" watchObservedRunningTime="2026-11-03 00:59:27.25667366 +0000 UTC m=+1112.470417368"
Nov 03 00:59:28.329316 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.329278    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 00:59:28.333619 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.333600    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.371508 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.371467    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 00:59:28.390831 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.390798    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.391054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.391020    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.391234 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.391108    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv57\" (UniqueName: \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.391319 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.391253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.433485 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.433425    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-db5422-serving-cert\""
Nov 03 00:59:28.492670 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.492634    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.492836 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.492749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.492888 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.492874    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.492951 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.492937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv57\" (UniqueName: \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.493582 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.493552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.495218 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.495191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.495543 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.495526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.509818 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.509794    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv57\" (UniqueName: \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\") pod \"maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.654262 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.654106    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:28.961722 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:28.961628    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 00:59:28.962346 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:59:28.962313    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829edc33_8113_4fe5_b0a2_264ad30d7b17.slice/crio-d30deab721fa6c7645b08d87740b72a713089b6f6d3128aa7545454fda4dfeb5 WatchSource:0}: Error finding container d30deab721fa6c7645b08d87740b72a713089b6f6d3128aa7545454fda4dfeb5: Status 404 returned error can't find the container with id d30deab721fa6c7645b08d87740b72a713089b6f6d3128aa7545454fda4dfeb5
Nov 03 00:59:29.247812 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:29.247715    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" event={"ID":"829edc33-8113-4fe5-b0a2-264ad30d7b17","Type":"ContainerStarted","Data":"d30deab721fa6c7645b08d87740b72a713089b6f6d3128aa7545454fda4dfeb5"}
Nov 03 00:59:30.214902 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:30.214867    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 00:59:30.273054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:30.271738    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" event={"ID":"829edc33-8113-4fe5-b0a2-264ad30d7b17","Type":"ContainerStarted","Data":"9745664c58fb0c26c2fcc979c5ac0b3ec99b8343de93b725ad8565be9280fa6f"}
Nov 03 00:59:30.273054 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:30.272792    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:30.295652 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:30.295588    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" podStartSLOduration=1.869589413 podStartE2EDuration="2.295568671s" totalImagesPullingTime="425.979258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:30.291449737 +0000 UTC m=+1115.505193410" watchObservedRunningTime="2026-11-03 00:59:30.295568671 +0000 UTC m=+1115.509312345"
Nov 03 00:59:33.245224 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:33.245195    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 00:59:37.297292 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:37.297217    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 00:59:39.271510 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.271468    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 00:59:39.276740 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.276706    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.281557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.281533    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-def754-serving-cert\""
Nov 03 00:59:39.285988 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.285925    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 00:59:39.451933 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.451903    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.452136 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.451950    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.452243 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.452214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.452332 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.452316    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.554254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.554131    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.554254 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.554251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.554490 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.554393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.554490 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.554438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.554970 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.554947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.557466 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.557437    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.557576 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.557483    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.565043 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.565011    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\") pod \"maas-api-e2e-ipp-b-def754-7657948bfd-clr4l\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.597322 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.597278    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:39.880952 ip-10-0-136-112 kubenswrapper[4531]: E1103 00:59:39.880916    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 00:59:39.918742 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:39.918707    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 00:59:40.365863 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:40.365809    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" event={"ID":"5822f180-d2bd-4d29-ba4c-a607f91c3c27","Type":"ContainerStarted","Data":"3b7e8894ecc3d3e72ef9d6f8f81a0ce5df6dcf28d2ae1057e9dc9585dee0d95d"}
Nov 03 00:59:41.375184 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:41.374495    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" event={"ID":"5822f180-d2bd-4d29-ba4c-a607f91c3c27","Type":"ContainerStarted","Data":"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"}
Nov 03 00:59:41.393309 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:41.393274    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:41.424110 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:41.424043    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" podStartSLOduration=1.916403409 podStartE2EDuration="2.424005381s" totalImagesPullingTime="507.601972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:41.419541023 +0000 UTC m=+1126.633284690" watchObservedRunningTime="2026-11-03 00:59:41.424005381 +0000 UTC m=+1126.637749057"
Nov 03 00:59:47.392481 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:47.392446    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 00:59:58.535564 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.535521    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 00:59:58.543050 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.543028    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 00:59:58.543200 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.543127    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.550994 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.550965    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ffdea6-serving-cert\""
Nov 03 00:59:58.698836 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.698789    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.699106 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.699076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.699197 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.699123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.699197 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.699150    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2z4\" (UniqueName: \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.800401 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.800317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.800557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.800464    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.800557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.800482    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.800557 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.800498    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2z4\" (UniqueName: \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.801193 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.801143    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.802942 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.802922    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.803102 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.803076    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.808500 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.808477    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2z4\" (UniqueName: \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\") pod \"maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:58.873859 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:58.873795    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 00:59:59.168148 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:59.168111    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 00:59:59.168669 ip-10-0-136-112 kubenswrapper[4531]: W1103 00:59:59.168632    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb35e80_259b_4d6c_be1a_664701799e41.slice/crio-938f26e4599387c5f1414251f4dcb11fedb03e7dc7c0cb8feb71b1b2095438d8 WatchSource:0}: Error finding container 938f26e4599387c5f1414251f4dcb11fedb03e7dc7c0cb8feb71b1b2095438d8: Status 404 returned error can't find the container with id 938f26e4599387c5f1414251f4dcb11fedb03e7dc7c0cb8feb71b1b2095438d8
Nov 03 00:59:59.594320 ip-10-0-136-112 kubenswrapper[4531]: I1103 00:59:59.594287    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" event={"ID":"fdb35e80-259b-4d6c-be1a-664701799e41","Type":"ContainerStarted","Data":"938f26e4599387c5f1414251f4dcb11fedb03e7dc7c0cb8feb71b1b2095438d8"}
Nov 03 01:00:00.603439 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:00.603401    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" event={"ID":"fdb35e80-259b-4d6c-be1a-664701799e41","Type":"ContainerStarted","Data":"01c0681c9096f183d4e1932e2cabeea14ac1044e95f46f86066e68b2f377c20b"}
Nov 03 01:00:00.603857 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:00.603492    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 01:00:00.627602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:00.627554    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" podStartSLOduration=1.860368714 podStartE2EDuration="2.627540745s" totalImagesPullingTime="767.172031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:00.624068152 +0000 UTC m=+1145.837811827" watchObservedRunningTime="2026-11-03 01:00:00.627540745 +0000 UTC m=+1145.841284416"
Nov 03 01:00:06.617009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:06.616976    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 01:00:09.887817 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:00:09.887778    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:00:23.896591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:23.896552    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 01:00:23.897090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:23.896823    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" podUID="d336b05a-dab8-4523-8e64-25d71ff87726" containerName="maas-api" containerID="cri-o://9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87" gracePeriod=30
Nov 03 01:00:24.394733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.394700    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 01:00:24.438006 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.437974    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\") pod \"d336b05a-dab8-4523-8e64-25d71ff87726\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") "
Nov 03 01:00:24.438211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.438055    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\") pod \"d336b05a-dab8-4523-8e64-25d71ff87726\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") "
Nov 03 01:00:24.438211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.438109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\" (UniqueName: \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\") pod \"d336b05a-dab8-4523-8e64-25d71ff87726\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") "
Nov 03 01:00:24.438340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.438281    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\") pod \"d336b05a-dab8-4523-8e64-25d71ff87726\" (UID: \"d336b05a-dab8-4523-8e64-25d71ff87726\") "
Nov 03 01:00:24.440629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.440492    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca" pod "d336b05a-dab8-4523-8e64-25d71ff87726" (UID: "d336b05a-dab8-4523-8e64-25d71ff87726"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:24.444358 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.444318    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc" pod "d336b05a-dab8-4523-8e64-25d71ff87726" (UID: "d336b05a-dab8-4523-8e64-25d71ff87726"). InnerVolumeSpecName "kube-api-access-lmttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.444560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.444543    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs" pod "d336b05a-dab8-4523-8e64-25d71ff87726" (UID: "d336b05a-dab8-4523-8e64-25d71ff87726"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:24.450571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.450530    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls" pod "d336b05a-dab8-4523-8e64-25d71ff87726" (UID: "d336b05a-dab8-4523-8e64-25d71ff87726"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:24.540978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.540940    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d336b05a-dab8-4523-8e64-25d71ff87726-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:24.540978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.540976    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:24.541196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.540989    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336b05a-dab8-4523-8e64-25d71ff87726-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:24.541196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.540998    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmttc\" (UniqueName: \"kubernetes.io/projected/d336b05a-dab8-4523-8e64-25d71ff87726-kube-api-access-lmttc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:24.845871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.845838    4531 generic.go:356] "Generic (PLEG): container finished" podID="d336b05a-dab8-4523-8e64-25d71ff87726" containerID="9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87" exitCode=0
Nov 03 01:00:24.846073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.845895    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" event={"ID":"d336b05a-dab8-4523-8e64-25d71ff87726","Type":"ContainerDied","Data":"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"}
Nov 03 01:00:24.846073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.845906    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"
Nov 03 01:00:24.846073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.845945    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz" event={"ID":"d336b05a-dab8-4523-8e64-25d71ff87726","Type":"ContainerDied","Data":"371341a812fb4d081941452a51f2fb07123721101c40e88827395a458b88c549"}
Nov 03 01:00:24.846073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.845971    4531 scope.go:122] "RemoveContainer" containerID="9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"
Nov 03 01:00:24.858188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.858147    4531 scope.go:122] "RemoveContainer" containerID="9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"
Nov 03 01:00:24.858527 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:00:24.858505    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87\": container with ID starting with 9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87 not found: ID does not exist" containerID="9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"
Nov 03 01:00:24.858608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.858535    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87"} err="failed to get container status \"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87\": rpc error: code = NotFound desc = could not find container \"9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87\": container with ID starting with 9e82ea25d2ab445cfb72a005094583800972f795fdeb9756dc1a4776b8a21b87 not found: ID does not exist"
Nov 03 01:00:24.875297 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.875237    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 01:00:24.879342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.879322    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47369d-544b548f99-2hzjz"]
Nov 03 01:00:24.905218 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:24.905186    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d336b05a-dab8-4523-8e64-25d71ff87726" path="/var/lib/kubelet/pods/d336b05a-dab8-4523-8e64-25d71ff87726/volumes"
Nov 03 01:00:37.565765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:37.548394    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 01:00:37.565765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:37.548674    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" podUID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" containerName="maas-api" containerID="cri-o://d732ca02e131741a7b44c99779037045ceb6ec6ac56622e0e763df1a87c85945" gracePeriod=30
Nov 03 01:00:38.031484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.030566    4531 generic.go:356] "Generic (PLEG): container finished" podID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" containerID="d732ca02e131741a7b44c99779037045ceb6ec6ac56622e0e763df1a87c85945" exitCode=0
Nov 03 01:00:38.031484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.030687    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" event={"ID":"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f","Type":"ContainerDied","Data":"d732ca02e131741a7b44c99779037045ceb6ec6ac56622e0e763df1a87c85945"}
Nov 03 01:00:38.031484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.030756    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj" event={"ID":"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f","Type":"ContainerDied","Data":"e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f"}
Nov 03 01:00:38.031484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.030799    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89fb417a95435e1bb59c6c53db4b55944023aec5072a111adbc91902c874d8f"
Nov 03 01:00:38.054230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.053350    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 01:00:38.273654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.264635    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\") pod \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") "
Nov 03 01:00:38.273654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.264726    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\") pod \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") "
Nov 03 01:00:38.273654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.264809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\" (UniqueName: \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\") pod \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") "
Nov 03 01:00:38.273654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.264840    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\") pod \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\" (UID: \"9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f\") "
Nov 03 01:00:38.273654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.266528    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca" pod "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" (UID: "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:38.274717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.274681    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv" pod "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" (UID: "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f"). InnerVolumeSpecName "kube-api-access-jmvwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:38.279792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.277675    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs" pod "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" (UID: "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:38.282250 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.281386    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls" pod "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" (UID: "9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:38.367444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.367021    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvwv\" (UniqueName: \"kubernetes.io/projected/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-kube-api-access-jmvwv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:38.367444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.367061    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:38.367444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.367084    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:38.367444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:38.367098    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:39.040807 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:39.040770    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"
Nov 03 01:00:39.073451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:39.073419    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 01:00:39.081906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:39.081865    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7406a9-56bc6db5cd-s9xrj"]
Nov 03 01:00:39.881519 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:00:39.881485    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:00:40.904625 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:40.904388    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" path="/var/lib/kubelet/pods/9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f/volumes"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.540946    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.542032    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" containerName="maas-api"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.542053    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" containerName="maas-api"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.542067    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d336b05a-dab8-4523-8e64-25d71ff87726" containerName="maas-api"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.542075    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336b05a-dab8-4523-8e64-25d71ff87726" containerName="maas-api"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.543354    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5dcbf9-3eab-4d3e-8b1e-fbddc9ce3a2f" containerName="maas-api"
Nov 03 01:00:47.546185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.543377    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336b05a-dab8-4523-8e64-25d71ff87726" containerName="maas-api"
Nov 03 01:00:47.548214 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.547406    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.553420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.552808    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1e352e-serving-cert\""
Nov 03 01:00:47.574056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.564495    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:00:47.639677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.639638    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.639924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.639901    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.639982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.639961    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flnb6\" (UniqueName: \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.640032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.640016    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.741881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.741845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.742077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.741909    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flnb6\" (UniqueName: \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.742077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.741958    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.742238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.742098    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.742666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.742642    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.744434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.744412    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.744533 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.744445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.751099 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.751074    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flnb6\" (UniqueName: \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\") pod \"maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:47.872815 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:47.872770    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:48.175209 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:00:48.174370    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55676db4_a628_419c_aa2b_f8b59179361d.slice/crio-fb5ec220fd456da1f6627c5aac3f634d86a66a2af3d38512d68ed89d08201fd0 WatchSource:0}: Error finding container fb5ec220fd456da1f6627c5aac3f634d86a66a2af3d38512d68ed89d08201fd0: Status 404 returned error can't find the container with id fb5ec220fd456da1f6627c5aac3f634d86a66a2af3d38512d68ed89d08201fd0
Nov 03 01:00:48.175209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:48.175038    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:00:49.131551 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:49.131499    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" event={"ID":"55676db4-a628-419c-aa2b-f8b59179361d","Type":"ContainerStarted","Data":"e71bf5e755f0ac48c5f3e8604ecf13949832d2e83f592f5bf69c549aed70fb2a"}
Nov 03 01:00:49.132049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:49.131612    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" event={"ID":"55676db4-a628-419c-aa2b-f8b59179361d","Type":"ContainerStarted","Data":"fb5ec220fd456da1f6627c5aac3f634d86a66a2af3d38512d68ed89d08201fd0"}
Nov 03 01:00:49.132049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:49.131690    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:49.152298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:49.152145    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" podStartSLOduration=1.581312589 podStartE2EDuration="2.152127308s" totalImagesPullingTime="570.814719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:49.149249252 +0000 UTC m=+1194.362992934" watchObservedRunningTime="2026-11-03 01:00:49.152127308 +0000 UTC m=+1194.365870980"
Nov 03 01:00:55.204220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:55.197342    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:00:56.128096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.128062    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 01:00:56.128631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.128583    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" containerName="maas-api" containerID="cri-o://01c0681c9096f183d4e1932e2cabeea14ac1044e95f46f86066e68b2f377c20b" gracePeriod=30
Nov 03 01:00:56.257327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.257245    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" event={"ID":"fdb35e80-259b-4d6c-be1a-664701799e41","Type":"ContainerDied","Data":"01c0681c9096f183d4e1932e2cabeea14ac1044e95f46f86066e68b2f377c20b"}
Nov 03 01:00:56.257327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.257298    4531 generic.go:356] "Generic (PLEG): container finished" podID="fdb35e80-259b-4d6c-be1a-664701799e41" containerID="01c0681c9096f183d4e1932e2cabeea14ac1044e95f46f86066e68b2f377c20b" exitCode=0
Nov 03 01:00:56.711365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.711340    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 01:00:56.805508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.805461    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\" (UniqueName: \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\") pod \"fdb35e80-259b-4d6c-be1a-664701799e41\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") "
Nov 03 01:00:56.805697 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.805519    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\") pod \"fdb35e80-259b-4d6c-be1a-664701799e41\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") "
Nov 03 01:00:56.805697 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.805643    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\") pod \"fdb35e80-259b-4d6c-be1a-664701799e41\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") "
Nov 03 01:00:56.805886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.805866    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\") pod \"fdb35e80-259b-4d6c-be1a-664701799e41\" (UID: \"fdb35e80-259b-4d6c-be1a-664701799e41\") "
Nov 03 01:00:56.805974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.805895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca" pod "fdb35e80-259b-4d6c-be1a-664701799e41" (UID: "fdb35e80-259b-4d6c-be1a-664701799e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:56.806450 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.806429    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb35e80-259b-4d6c-be1a-664701799e41-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:56.808001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.807972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs" pod "fdb35e80-259b-4d6c-be1a-664701799e41" (UID: "fdb35e80-259b-4d6c-be1a-664701799e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:56.808193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.808153    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4" pod "fdb35e80-259b-4d6c-be1a-664701799e41" (UID: "fdb35e80-259b-4d6c-be1a-664701799e41"). InnerVolumeSpecName "kube-api-access-xb2z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:56.808301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.808233    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls" pod "fdb35e80-259b-4d6c-be1a-664701799e41" (UID: "fdb35e80-259b-4d6c-be1a-664701799e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:56.907852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.907814    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2z4\" (UniqueName: \"kubernetes.io/projected/fdb35e80-259b-4d6c-be1a-664701799e41-kube-api-access-xb2z4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:56.907852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.907851    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:56.908067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:56.907867    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb35e80-259b-4d6c-be1a-664701799e41-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:00:57.268047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:57.268014    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" event={"ID":"fdb35e80-259b-4d6c-be1a-664701799e41","Type":"ContainerDied","Data":"938f26e4599387c5f1414251f4dcb11fedb03e7dc7c0cb8feb71b1b2095438d8"}
Nov 03 01:00:57.268540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:57.268079    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"
Nov 03 01:00:57.268540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:57.268101    4531 scope.go:122] "RemoveContainer" containerID="01c0681c9096f183d4e1932e2cabeea14ac1044e95f46f86066e68b2f377c20b"
Nov 03 01:00:57.288947 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:57.288921    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 01:00:57.292636 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:57.292614    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2"]
Nov 03 01:00:58.907483 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:58.907440    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" path="/var/lib/kubelet/pods/fdb35e80-259b-4d6c-be1a-664701799e41/volumes"
Nov 03 01:00:59.632184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:00:59.628271    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ffdea6-7d4d594566-nzmc2" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.91:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:01:05.480138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:05.480095    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 01:01:05.480674 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:05.480396    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" podUID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" containerName="maas-api" containerID="cri-o://aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e" gracePeriod=30
Nov 03 01:01:06.111259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.111222    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 01:01:06.210050 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.210014    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\") pod \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") "
Nov 03 01:01:06.210268 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.210104    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\") pod \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") "
Nov 03 01:01:06.210357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.210338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\") pod \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") "
Nov 03 01:01:06.210458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.210441    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\") pod \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\" (UID: \"5822f180-d2bd-4d29-ba4c-a607f91c3c27\") "
Nov 03 01:01:06.210887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.210844    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca" pod "5822f180-d2bd-4d29-ba4c-a607f91c3c27" (UID: "5822f180-d2bd-4d29-ba4c-a607f91c3c27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:06.211186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.211137    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5822f180-d2bd-4d29-ba4c-a607f91c3c27-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:01:06.212376 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.212354    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs" pod "5822f180-d2bd-4d29-ba4c-a607f91c3c27" (UID: "5822f180-d2bd-4d29-ba4c-a607f91c3c27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:06.212705 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.212681    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls" pod "5822f180-d2bd-4d29-ba4c-a607f91c3c27" (UID: "5822f180-d2bd-4d29-ba4c-a607f91c3c27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:06.212758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.212684    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs" pod "5822f180-d2bd-4d29-ba4c-a607f91c3c27" (UID: "5822f180-d2bd-4d29-ba4c-a607f91c3c27"). InnerVolumeSpecName "kube-api-access-sdqcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:06.312469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.312437    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:01:06.312469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.312468    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqcs\" (UniqueName: \"kubernetes.io/projected/5822f180-d2bd-4d29-ba4c-a607f91c3c27-kube-api-access-sdqcs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:01:06.312469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.312481    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5822f180-d2bd-4d29-ba4c-a607f91c3c27-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:01:06.400976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.400877    4531 generic.go:356] "Generic (PLEG): container finished" podID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" containerID="aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e" exitCode=0
Nov 03 01:01:06.400976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.400938    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"
Nov 03 01:01:06.401256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.400974    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" event={"ID":"5822f180-d2bd-4d29-ba4c-a607f91c3c27","Type":"ContainerDied","Data":"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"}
Nov 03 01:01:06.401256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.401081    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l" event={"ID":"5822f180-d2bd-4d29-ba4c-a607f91c3c27","Type":"ContainerDied","Data":"3b7e8894ecc3d3e72ef9d6f8f81a0ce5df6dcf28d2ae1057e9dc9585dee0d95d"}
Nov 03 01:01:06.401256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.401116    4531 scope.go:122] "RemoveContainer" containerID="aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"
Nov 03 01:01:06.413571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.413468    4531 scope.go:122] "RemoveContainer" containerID="aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"
Nov 03 01:01:06.413967 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:01:06.413946    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e\": container with ID starting with aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e not found: ID does not exist" containerID="aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"
Nov 03 01:01:06.414061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.413979    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e"} err="failed to get container status \"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e\": rpc error: code = NotFound desc = could not find container \"aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e\": container with ID starting with aa770dc6320d4893b5a6dd99d128c8fdfdb7f8a076aacfc7acc27906d947ed1e not found: ID does not exist"
Nov 03 01:01:06.449286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.448986    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 01:01:06.455062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.455033    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-def754-7657948bfd-clr4l"]
Nov 03 01:01:06.904865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:06.904826    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" path="/var/lib/kubelet/pods/5822f180-d2bd-4d29-ba4c-a607f91c3c27/volumes"
Nov 03 01:01:10.031723 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:01:10.031687    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:01:21.620152 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.620111    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:01:21.621052 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621028    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" containerName="maas-api"
Nov 03 01:01:21.621213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621053    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" containerName="maas-api"
Nov 03 01:01:21.621213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621122    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" containerName="maas-api"
Nov 03 01:01:21.621213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621130    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" containerName="maas-api"
Nov 03 01:01:21.622845 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621291    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5822f180-d2bd-4d29-ba4c-a607f91c3c27" containerName="maas-api"
Nov 03 01:01:21.622845 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.621309    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb35e80-259b-4d6c-be1a-664701799e41" containerName="maas-api"
Nov 03 01:01:21.625827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.625807    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.646407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.646373    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:01:21.668301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.668269    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-208a0d-serving-cert\""
Nov 03 01:01:21.774189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.773931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.774189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.774147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.774546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.774201    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttbt\" (UniqueName: \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.774546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.774322    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.875588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.875501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.875588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.875547    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttbt\" (UniqueName: \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.875836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.875661    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.875836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.875733    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.876575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.876545    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.878242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.878212    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.878329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.878259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.884220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.884198    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttbt\" (UniqueName: \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\") pod \"maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:21.955153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:21.955121    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:22.249410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:22.249382    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:01:22.810538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:22.810497    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" event={"ID":"01ecadde-db6c-4c61-979d-b4e282e3ff4f","Type":"ContainerStarted","Data":"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"}
Nov 03 01:01:22.810920 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:22.810555    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" event={"ID":"01ecadde-db6c-4c61-979d-b4e282e3ff4f","Type":"ContainerStarted","Data":"9b57c0526ccb224b9b3f6fa4b9845eaab890d6be9521d9ea000cacc18ec8810e"}
Nov 03 01:01:22.811293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:22.811269    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:22.831687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:22.831637    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" podStartSLOduration=1.400495494 podStartE2EDuration="1.831623148s" totalImagesPullingTime="431.127654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:22.828598527 +0000 UTC m=+1228.042342194" watchObservedRunningTime="2026-11-03 01:01:22.831623148 +0000 UTC m=+1228.045366819"
Nov 03 01:01:29.830797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:29.830770    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:01:37.799361 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.799319    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-th6d7-25rth"]
Nov 03 01:01:37.803440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.803417    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:37.807556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.807533    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 01:01:37.811820 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.811792    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-th6d7-25rth"]
Nov 03 01:01:37.811963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.811927    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 01:01:37.811963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.811955    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tg82s\""
Nov 03 01:01:37.969867 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.969833    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmxt\" (UniqueName: \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:37.970040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.969892    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:37.970142 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.970117    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:37.970240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:37.970221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.071892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.071863    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072074 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmxt\" (UniqueName: \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072146 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.072936 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.072914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.080691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.080670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmxt\" (UniqueName: \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\") pod \"periodic-gathering-th6d7-25rth\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") " pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.122671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.122645    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:01:38.410463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.410432    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-th6d7-25rth"]
Nov 03 01:01:38.410992 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:01:38.410959    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e4978e_91c9_4c21_a10a_aa554ca132ec.slice/crio-2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c WatchSource:0}: Error finding container 2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c: Status 404 returned error can't find the container with id 2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c
Nov 03 01:01:38.947036 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.946993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-th6d7-25rth" event={"ID":"89e4978e-91c9-4c21-a10a-aa554ca132ec","Type":"ContainerStarted","Data":"b6069dd5e831cb784a0d4c79e4edf0920a7edfe69129cf190bfe4f7e8c04b5b8"}
Nov 03 01:01:38.947550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.947044    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-th6d7-25rth" event={"ID":"89e4978e-91c9-4c21-a10a-aa554ca132ec","Type":"ContainerStarted","Data":"2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c"}
Nov 03 01:01:38.967682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:01:38.967615    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-th6d7-25rth" podStartSLOduration=1.967596194 podStartE2EDuration="1.967596194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:38.96560708 +0000 UTC m=+1244.179350752" watchObservedRunningTime="2026-11-03 01:01:38.967596194 +0000 UTC m=+1244.181339867"
Nov 03 01:01:39.897110 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:01:39.897067    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:02:09.929858 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:02:09.929815    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:02:39.881590 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:02:39.881465    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:02:49.386212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:49.375255    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 01:02:49.386212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:49.375650    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" podUID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" containerName="maas-api" containerID="cri-o://ba5b4a4eafa03ca81721790d599fae587a024fac8a757f29f707c9a87db49888" gracePeriod=30
Nov 03 01:02:49.814108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:49.813756    4531 generic.go:356] "Generic (PLEG): container finished" podID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" containerID="ba5b4a4eafa03ca81721790d599fae587a024fac8a757f29f707c9a87db49888" exitCode=0
Nov 03 01:02:49.814108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:49.813951    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" event={"ID":"2334dcb7-06f3-4801-ac01-fff02d1e4fea","Type":"ContainerDied","Data":"ba5b4a4eafa03ca81721790d599fae587a024fac8a757f29f707c9a87db49888"}
Nov 03 01:02:50.064544 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.064511    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 01:02:50.169780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.169735    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\") pod \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") "
Nov 03 01:02:50.170004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.169932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\" (UniqueName: \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\") pod \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") "
Nov 03 01:02:50.170100 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.170080    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\") pod \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") "
Nov 03 01:02:50.170193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.170119    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\") pod \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\" (UID: \"2334dcb7-06f3-4801-ac01-fff02d1e4fea\") "
Nov 03 01:02:50.170652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.170601    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca" pod "2334dcb7-06f3-4801-ac01-fff02d1e4fea" (UID: "2334dcb7-06f3-4801-ac01-fff02d1e4fea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:02:50.170990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.170961    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2334dcb7-06f3-4801-ac01-fff02d1e4fea-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:02:50.173111 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.173065    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs" pod "2334dcb7-06f3-4801-ac01-fff02d1e4fea" (UID: "2334dcb7-06f3-4801-ac01-fff02d1e4fea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:50.173378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.173348    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls" pod "2334dcb7-06f3-4801-ac01-fff02d1e4fea" (UID: "2334dcb7-06f3-4801-ac01-fff02d1e4fea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:50.173564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.173537    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb" pod "2334dcb7-06f3-4801-ac01-fff02d1e4fea" (UID: "2334dcb7-06f3-4801-ac01-fff02d1e4fea"). InnerVolumeSpecName "kube-api-access-vmxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:50.272967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.272876    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:02:50.272967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.272906    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2334dcb7-06f3-4801-ac01-fff02d1e4fea-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:02:50.272967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.272916    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmxnb\" (UniqueName: \"kubernetes.io/projected/2334dcb7-06f3-4801-ac01-fff02d1e4fea-kube-api-access-vmxnb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:02:50.822658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.822629    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"
Nov 03 01:02:50.823062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.822628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6" event={"ID":"2334dcb7-06f3-4801-ac01-fff02d1e4fea","Type":"ContainerDied","Data":"f80af7b0dc578ac83fa1055027420d745ff11c7f67d0f92fc8e8a02c451ee303"}
Nov 03 01:02:50.823062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.822764    4531 scope.go:122] "RemoveContainer" containerID="ba5b4a4eafa03ca81721790d599fae587a024fac8a757f29f707c9a87db49888"
Nov 03 01:02:50.853982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.853944    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 01:02:50.858201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.858174    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-07f5f6-84f6d4d67b-g5kt6"]
Nov 03 01:02:50.903643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:02:50.903613    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" path="/var/lib/kubelet/pods/2334dcb7-06f3-4801-ac01-fff02d1e4fea/volumes"
Nov 03 01:03:04.738692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:04.738652    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 01:03:04.739203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:04.738863    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" podUID="829edc33-8113-4fe5-b0a2-264ad30d7b17" containerName="maas-api" containerID="cri-o://9745664c58fb0c26c2fcc979c5ac0b3ec99b8343de93b725ad8565be9280fa6f" gracePeriod=30
Nov 03 01:03:04.996388 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:04.996305    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" event={"ID":"829edc33-8113-4fe5-b0a2-264ad30d7b17","Type":"ContainerDied","Data":"9745664c58fb0c26c2fcc979c5ac0b3ec99b8343de93b725ad8565be9280fa6f"}
Nov 03 01:03:04.996600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:04.996307    4531 generic.go:356] "Generic (PLEG): container finished" podID="829edc33-8113-4fe5-b0a2-264ad30d7b17" containerID="9745664c58fb0c26c2fcc979c5ac0b3ec99b8343de93b725ad8565be9280fa6f" exitCode=0
Nov 03 01:03:05.318056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.318028    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 01:03:05.510847 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.510786    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\") pod \"829edc33-8113-4fe5-b0a2-264ad30d7b17\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") "
Nov 03 01:03:05.511037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.510938    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\") pod \"829edc33-8113-4fe5-b0a2-264ad30d7b17\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") "
Nov 03 01:03:05.511037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.510985    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\" (UniqueName: \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\") pod \"829edc33-8113-4fe5-b0a2-264ad30d7b17\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") "
Nov 03 01:03:05.511260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.511231    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca" pod "829edc33-8113-4fe5-b0a2-264ad30d7b17" (UID: "829edc33-8113-4fe5-b0a2-264ad30d7b17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:05.511364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.511287    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\") pod \"829edc33-8113-4fe5-b0a2-264ad30d7b17\" (UID: \"829edc33-8113-4fe5-b0a2-264ad30d7b17\") "
Nov 03 01:03:05.511941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.511916    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829edc33-8113-4fe5-b0a2-264ad30d7b17-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:05.514048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.514016    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs" pod "829edc33-8113-4fe5-b0a2-264ad30d7b17" (UID: "829edc33-8113-4fe5-b0a2-264ad30d7b17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:03:05.514048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.514032    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57" pod "829edc33-8113-4fe5-b0a2-264ad30d7b17" (UID: "829edc33-8113-4fe5-b0a2-264ad30d7b17"). InnerVolumeSpecName "kube-api-access-rfv57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:05.514712 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.514689    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls" pod "829edc33-8113-4fe5-b0a2-264ad30d7b17" (UID: "829edc33-8113-4fe5-b0a2-264ad30d7b17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:03:05.622183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.621735    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:05.622183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.621778    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829edc33-8113-4fe5-b0a2-264ad30d7b17-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:05.622183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:05.621794    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfv57\" (UniqueName: \"kubernetes.io/projected/829edc33-8113-4fe5-b0a2-264ad30d7b17-kube-api-access-rfv57\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:06.006719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.006643    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"
Nov 03 01:03:06.006719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.006683    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8" event={"ID":"829edc33-8113-4fe5-b0a2-264ad30d7b17","Type":"ContainerDied","Data":"d30deab721fa6c7645b08d87740b72a713089b6f6d3128aa7545454fda4dfeb5"}
Nov 03 01:03:06.007244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.006763    4531 scope.go:122] "RemoveContainer" containerID="9745664c58fb0c26c2fcc979c5ac0b3ec99b8343de93b725ad8565be9280fa6f"
Nov 03 01:03:06.049359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.049329    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 01:03:06.067049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.067014    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db5422-686f646cdd-gmlv8"]
Nov 03 01:03:06.975323 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:06.975275    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829edc33-8113-4fe5-b0a2-264ad30d7b17" path="/var/lib/kubelet/pods/829edc33-8113-4fe5-b0a2-264ad30d7b17/volumes"
Nov 03 01:03:09.879308 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:03:09.879267    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:03:14.084339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:14.084306    4531 generic.go:356] "Generic (PLEG): container finished" podID="89e4978e-91c9-4c21-a10a-aa554ca132ec" containerID="b6069dd5e831cb784a0d4c79e4edf0920a7edfe69129cf190bfe4f7e8c04b5b8" exitCode=0
Nov 03 01:03:14.084737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:14.084373    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-th6d7-25rth" event={"ID":"89e4978e-91c9-4c21-a10a-aa554ca132ec","Type":"ContainerDied","Data":"b6069dd5e831cb784a0d4c79e4edf0920a7edfe69129cf190bfe4f7e8c04b5b8"}
Nov 03 01:03:15.281441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.281400    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:03:15.330692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.330656    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\") pod \"89e4978e-91c9-4c21-a10a-aa554ca132ec\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") "
Nov 03 01:03:15.330913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.330738    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\") pod \"89e4978e-91c9-4c21-a10a-aa554ca132ec\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") "
Nov 03 01:03:15.330913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.330783    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\" (UniqueName: \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\") pod \"89e4978e-91c9-4c21-a10a-aa554ca132ec\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") "
Nov 03 01:03:15.331255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.331025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\") pod \"89e4978e-91c9-4c21-a10a-aa554ca132ec\" (UID: \"89e4978e-91c9-4c21-a10a-aa554ca132ec\") "
Nov 03 01:03:15.332035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.331787    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle" pod "89e4978e-91c9-4c21-a10a-aa554ca132ec" (UID: "89e4978e-91c9-4c21-a10a-aa554ca132ec"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:15.332035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.331814    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle" pod "89e4978e-91c9-4c21-a10a-aa554ca132ec" (UID: "89e4978e-91c9-4c21-a10a-aa554ca132ec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:15.332225 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.332059    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-service-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:15.332225 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.332078    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e4978e-91c9-4c21-a10a-aa554ca132ec-trusted-ca-bundle\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:15.332550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.332492    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path" pod "89e4978e-91c9-4c21-a10a-aa554ca132ec" (UID: "89e4978e-91c9-4c21-a10a-aa554ca132ec"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:15.334849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.334813    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt" pod "89e4978e-91c9-4c21-a10a-aa554ca132ec" (UID: "89e4978e-91c9-4c21-a10a-aa554ca132ec"). InnerVolumeSpecName "kube-api-access-lpmxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:15.433600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.433567    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89e4978e-91c9-4c21-a10a-aa554ca132ec-archives-path\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:15.433600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:15.433598    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpmxt\" (UniqueName: \"kubernetes.io/projected/89e4978e-91c9-4c21-a10a-aa554ca132ec-kube-api-access-lpmxt\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:03:16.149653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:16.148905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-th6d7-25rth" event={"ID":"89e4978e-91c9-4c21-a10a-aa554ca132ec","Type":"ContainerDied","Data":"2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c"}
Nov 03 01:03:16.149653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:16.148970    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fcfb25a95e9bbff44a670dce063e458be7625999b82298027ee1623bc62e96c"
Nov 03 01:03:16.149653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:03:16.149069    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-th6d7-25rth"
Nov 03 01:03:39.878686 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:03:39.878646    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:04:09.881524 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:04:09.881434    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:04:24.852185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:24.851663    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:04:24.852185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:24.851932    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" podUID="55676db4-a628-419c-aa2b-f8b59179361d" containerName="maas-api" containerID="cri-o://e71bf5e755f0ac48c5f3e8604ecf13949832d2e83f592f5bf69c549aed70fb2a" gracePeriod=30
Nov 03 01:04:25.193243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.181040    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" podUID="55676db4-a628-419c-aa2b-f8b59179361d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.92:8443/health\": dial tcp 10.133.0.92:8443: connect: connection refused"
Nov 03 01:04:25.307655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.300414    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 01:04:25.381570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.381514    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 01:04:25.438414 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.438370    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:25.928186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.927318    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 01:04:25.970183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:25.959248    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:26.280269 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.183676    4531 generic.go:356] "Generic (PLEG): container finished" podID="55676db4-a628-419c-aa2b-f8b59179361d" containerID="e71bf5e755f0ac48c5f3e8604ecf13949832d2e83f592f5bf69c549aed70fb2a" exitCode=0
Nov 03 01:04:26.460365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.185227    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" event={"ID":"55676db4-a628-419c-aa2b-f8b59179361d","Type":"ContainerDied","Data":"e71bf5e755f0ac48c5f3e8604ecf13949832d2e83f592f5bf69c549aed70fb2a"}
Nov 03 01:04:26.507184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.319413    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 01:04:26.553281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.552277    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:26.688185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.687263    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:26.932842 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:26.932796    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.000208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.000131    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.172234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.168311    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:04:27.172234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.168411    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.343289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.327599    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.484185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.436341    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 01:04:27.486912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.486885    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:04:27.540185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.533420    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 01:04:27.540185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.534152    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:04:27.872197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.863265    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.925230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.907553    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:27.925230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.908631    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:04:27.925230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.908677    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:28.021349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.927911    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:04:28.021349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.928014    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:04:28.021349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:27.971328    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 01:04:28.331546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.255584    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:04:28.331546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.255647    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:28.467690 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.467641    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 01:04:28.518239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.518192    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 01:04:28.562184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.538210    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:28.833184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.829370    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:04:28.868607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.867500    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 03 01:04:28.884025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.883993    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:04:28.884333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.884314    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:04:28.925366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:28.925337    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"
Nov 03 01:04:29.007382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.005964    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:04:29.007382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.006035    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:04:29.007382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.006052    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg" event={"ID":"55676db4-a628-419c-aa2b-f8b59179361d","Type":"ContainerDied","Data":"fb5ec220fd456da1f6627c5aac3f634d86a66a2af3d38512d68ed89d08201fd0"}
Nov 03 01:04:29.007382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.006102    4531 scope.go:122] "RemoveContainer" containerID="e71bf5e755f0ac48c5f3e8604ecf13949832d2e83f592f5bf69c549aed70fb2a"
Nov 03 01:04:29.023370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.021050    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\") pod \"55676db4-a628-419c-aa2b-f8b59179361d\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") "
Nov 03 01:04:29.023370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.021309    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\" (UniqueName: \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\") pod \"55676db4-a628-419c-aa2b-f8b59179361d\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") "
Nov 03 01:04:29.029317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.024854    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6" pod "55676db4-a628-419c-aa2b-f8b59179361d" (UID: "55676db4-a628-419c-aa2b-f8b59179361d"). InnerVolumeSpecName "kube-api-access-flnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:29.033042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.032946    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls" pod "55676db4-a628-419c-aa2b-f8b59179361d" (UID: "55676db4-a628-419c-aa2b-f8b59179361d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:29.123046 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.123010    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\") pod \"55676db4-a628-419c-aa2b-f8b59179361d\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") "
Nov 03 01:04:29.123248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.123225    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\") pod \"55676db4-a628-419c-aa2b-f8b59179361d\" (UID: \"55676db4-a628-419c-aa2b-f8b59179361d\") "
Nov 03 01:04:29.123443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.123415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca" pod "55676db4-a628-419c-aa2b-f8b59179361d" (UID: "55676db4-a628-419c-aa2b-f8b59179361d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:29.124378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.124349    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55676db4-a628-419c-aa2b-f8b59179361d-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:29.125732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.124385    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flnb6\" (UniqueName: \"kubernetes.io/projected/55676db4-a628-419c-aa2b-f8b59179361d-kube-api-access-flnb6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:29.125732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.124402    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:29.131510 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.131457    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs" pod "55676db4-a628-419c-aa2b-f8b59179361d" (UID: "55676db4-a628-419c-aa2b-f8b59179361d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:29.225745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.225712    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55676db4-a628-419c-aa2b-f8b59179361d-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:29.257566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.257506    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:04:29.263595 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:29.263565    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1e352e-99577cd6f-xzbwg"]
Nov 03 01:04:30.916088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:30.916052    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55676db4-a628-419c-aa2b-f8b59179361d" path="/var/lib/kubelet/pods/55676db4-a628-419c-aa2b-f8b59179361d/volumes"
Nov 03 01:04:39.884351 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:04:39.884305    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:04:41.083527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.083487    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:04:41.084054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.083761    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" podUID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" containerName="maas-api" containerID="cri-o://39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029" gracePeriod=30
Nov 03 01:04:41.637348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.637308    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:04:41.793614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.793534    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\") pod \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") "
Nov 03 01:04:41.793614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.793598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\" (UniqueName: \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\") pod \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") "
Nov 03 01:04:41.793797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.793711    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\") pod \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") "
Nov 03 01:04:41.793870 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.793849    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\") pod \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\" (UID: \"01ecadde-db6c-4c61-979d-b4e282e3ff4f\") "
Nov 03 01:04:41.794231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.794199    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca" pod "01ecadde-db6c-4c61-979d-b4e282e3ff4f" (UID: "01ecadde-db6c-4c61-979d-b4e282e3ff4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:41.794523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.794504    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ecadde-db6c-4c61-979d-b4e282e3ff4f-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:41.795763 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.795737    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls" pod "01ecadde-db6c-4c61-979d-b4e282e3ff4f" (UID: "01ecadde-db6c-4c61-979d-b4e282e3ff4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:41.796145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.796127    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs" pod "01ecadde-db6c-4c61-979d-b4e282e3ff4f" (UID: "01ecadde-db6c-4c61-979d-b4e282e3ff4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:41.796229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.796195    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt" pod "01ecadde-db6c-4c61-979d-b4e282e3ff4f" (UID: "01ecadde-db6c-4c61-979d-b4e282e3ff4f"). InnerVolumeSpecName "kube-api-access-xttbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:41.896065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.896029    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:41.896065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.896066    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ecadde-db6c-4c61-979d-b4e282e3ff4f-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:41.896337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:41.896082    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttbt\" (UniqueName: \"kubernetes.io/projected/01ecadde-db6c-4c61-979d-b4e282e3ff4f-kube-api-access-xttbt\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:42.043517 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.043485    4531 generic.go:356] "Generic (PLEG): container finished" podID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" containerID="39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029" exitCode=0
Nov 03 01:04:42.043733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.043553    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"
Nov 03 01:04:42.043733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.043576    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" event={"ID":"01ecadde-db6c-4c61-979d-b4e282e3ff4f","Type":"ContainerDied","Data":"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"}
Nov 03 01:04:42.043733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.043643    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk" event={"ID":"01ecadde-db6c-4c61-979d-b4e282e3ff4f","Type":"ContainerDied","Data":"9b57c0526ccb224b9b3f6fa4b9845eaab890d6be9521d9ea000cacc18ec8810e"}
Nov 03 01:04:42.043733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.043672    4531 scope.go:122] "RemoveContainer" containerID="39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"
Nov 03 01:04:42.057527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.057505    4531 scope.go:122] "RemoveContainer" containerID="39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"
Nov 03 01:04:42.057827 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:04:42.057807    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029\": container with ID starting with 39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029 not found: ID does not exist" containerID="39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"
Nov 03 01:04:42.057904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.057840    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029"} err="failed to get container status \"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029\": rpc error: code = NotFound desc = could not find container \"39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029\": container with ID starting with 39a5cc11303413beb0b7de27701119b74b65e7b38e1de2cb5ee093434d630029 not found: ID does not exist"
Nov 03 01:04:42.069986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.069961    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:04:42.073560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.073538    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-208a0d-db8fd546d-mx6mk"]
Nov 03 01:04:42.919180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:42.918514    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" path="/var/lib/kubelet/pods/01ecadde-db6c-4c61-979d-b4e282e3ff4f/volumes"
Nov 03 01:04:44.939187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:44.933297    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:45.061187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.044299    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 03 01:04:45.267910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.267807    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:45.406381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.406319    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:45.469203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.457608    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:45.469509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.465734    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 01:04:45.470849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.470784    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy" containerID="cri-o://1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076" gracePeriod=30
Nov 03 01:04:45.624216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:45.623602    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:04:50.719955 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.719930    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 01:04:50.815368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815275    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815371    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815394    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815420    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815457    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815597    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:50.815848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815679    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.815955 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815861    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:50.815955 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.815880    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\") pod \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\" (UID: \"0a553979-f84e-47b0-8f26-2b70ecbb8cbf\") "
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816220    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816295    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816401    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816711    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816734    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816751    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.816804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.816767    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.818244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.818214    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:04:50.818331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.818238    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:50.818331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.818238    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:50.818487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.818468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft" pod "0a553979-f84e-47b0-8f26-2b70ecbb8cbf" (UID: "0a553979-f84e-47b0-8f26-2b70ecbb8cbf"). InnerVolumeSpecName "kube-api-access-rbbft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:50.918384 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.918355    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbft\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-kube-api-access-rbbft\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.918384 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.918384    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.918546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.918398    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.918546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.918412    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:50.918546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:50.918424    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a553979-f84e-47b0-8f26-2b70ecbb8cbf-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:04:51.485475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.485438    4531 generic.go:356] "Generic (PLEG): container finished" podID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerID="1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076" exitCode=0
Nov 03 01:04:51.485672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.485494    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"
Nov 03 01:04:51.485672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.485513    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" event={"ID":"0a553979-f84e-47b0-8f26-2b70ecbb8cbf","Type":"ContainerDied","Data":"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"}
Nov 03 01:04:51.485672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.485568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs" event={"ID":"0a553979-f84e-47b0-8f26-2b70ecbb8cbf","Type":"ContainerDied","Data":"b5e7b6cef90463f636400a291e5e6f298bf3307a625ec0599ea55127b004e738"}
Nov 03 01:04:51.485672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.485590    4531 scope.go:122] "RemoveContainer" containerID="1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"
Nov 03 01:04:51.498699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.498657    4531 scope.go:122] "RemoveContainer" containerID="1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"
Nov 03 01:04:51.499043 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:04:51.499015    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076\": container with ID starting with 1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076 not found: ID does not exist" containerID="1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"
Nov 03 01:04:51.499109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.499054    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076"} err="failed to get container status \"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076\": rpc error: code = NotFound desc = could not find container \"1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076\": container with ID starting with 1da5f8b32b522ee0f76ec7206b3b89d0912e3d291bf099f1610d93743ff5d076 not found: ID does not exist"
Nov 03 01:04:51.509249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.509225    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 01:04:51.514898 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:51.514876    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-9xvbs"]
Nov 03 01:04:52.904769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:04:52.904725    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" path="/var/lib/kubelet/pods/0a553979-f84e-47b0-8f26-2b70ecbb8cbf/volumes"
Nov 03 01:05:03.871453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:03.871406    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:04.623308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:04.623257    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:04.920928 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:04.919268    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:04.935893 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:04.935796    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:04.978182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:04.952770    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:05:05.274285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.264338    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:05.368322 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.368274    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 01:05:05.626323 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.626274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:05.627027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.627007    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:05:05.920612 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.920514    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:05.951174 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.951118    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 01:05:05.952493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:05.952467    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:05:06.103134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.103105    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:05:06.299076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.298925    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:06.328249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.326135    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 01:05:06.840358 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.840308    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 01:05:06.932227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.921204    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:06.953409 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:06.952270    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 01:05:07.463624 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:07.453402    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 01:05:07.691836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:07.691805    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:05:07.878208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:07.869452    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 01:05:07.938546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:07.937291    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:07.959787 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:07.959745    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:05:08.238249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:08.235304    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:05:08.425441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:08.425406    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 01:05:08.468728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:08.468696    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:05:08.583355 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:08.580515    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:05:09.879485 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:09.879446    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:05:18.610204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:18.598764    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 01:05:18.610204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:18.607412    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy" containerID="cri-o://65abfb1c02afecb876507a843d04f8d4628e2b7ba00367b5129a3d21798725b1" gracePeriod=30
Nov 03 01:05:19.817289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.817255    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:05:19.819359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819335    4531 cpu_manager.go:414] "containerMap: removing container" podUID="829edc33-8113-4fe5-b0a2-264ad30d7b17" containerName="maas-api"
Nov 03 01:05:19.819359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819361    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829edc33-8113-4fe5-b0a2-264ad30d7b17" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819388    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819395    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819407    4531 cpu_manager.go:414] "containerMap: removing container" podUID="55676db4-a628-419c-aa2b-f8b59179361d" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819414    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55676db4-a628-419c-aa2b-f8b59179361d" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819430    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819436    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819511    4531 cpu_manager.go:414] "containerMap: removing container" podUID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819519    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" containerName="maas-api"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819534    4531 cpu_manager.go:414] "containerMap: removing container" podUID="89e4978e-91c9-4c21-a10a-aa554ca132ec" containerName="insights-gathering"
Nov 03 01:05:19.819557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819542    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e4978e-91c9-4c21-a10a-aa554ca132ec" containerName="insights-gathering"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819682    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ecadde-db6c-4c61-979d-b4e282e3ff4f" containerName="maas-api"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819694    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e4978e-91c9-4c21-a10a-aa554ca132ec" containerName="insights-gathering"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819707    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="829edc33-8113-4fe5-b0a2-264ad30d7b17" containerName="maas-api"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819718    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a553979-f84e-47b0-8f26-2b70ecbb8cbf" containerName="istio-proxy"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819727    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="55676db4-a628-419c-aa2b-f8b59179361d" containerName="maas-api"
Nov 03 01:05:19.820035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.819738    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2334dcb7-06f3-4801-ac01-fff02d1e4fea" containerName="maas-api"
Nov 03 01:05:19.823324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.823300    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:19.837013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.836968    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:05:19.843984 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.843841    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-4248ae-serving-cert\""
Nov 03 01:05:19.968187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.942209    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:19.968766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.968722    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzgk\" (UniqueName: \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:19.977183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.969100    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:19.977509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:19.977483    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.079687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.079647    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.079992 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.079973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.080140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.080127    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.080592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.080567    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzgk\" (UniqueName: \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.080853 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.080631    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.083628 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.083451    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.083628 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.083582    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.090931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.090904    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzgk\" (UniqueName: \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\") pod \"maas-api-e2e-autores-4248ae-77dddff8b6-8fs94\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.164291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.164254    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:20.469644 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:05:20.469601    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb51cb4_357a_45ef_9983_61c3125dfd5c.slice/crio-1263f8d88e2e150ba5843871b613ff800d7de4b5bde5dadfe364588bece59e22 WatchSource:0}: Error finding container 1263f8d88e2e150ba5843871b613ff800d7de4b5bde5dadfe364588bece59e22: Status 404 returned error can't find the container with id 1263f8d88e2e150ba5843871b613ff800d7de4b5bde5dadfe364588bece59e22
Nov 03 01:05:20.474143 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.473125    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:05:20.480188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:20.478099    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:05:21.174329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:21.174289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" event={"ID":"4fb51cb4-357a-45ef-9983-61c3125dfd5c","Type":"ContainerStarted","Data":"b0d7c239f3a76baaa820afca69fcb858b95d957ddf99ca9acbe454f4cdf59c77"}
Nov 03 01:05:21.174761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:21.174517    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:21.174761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:21.174547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" event={"ID":"4fb51cb4-357a-45ef-9983-61c3125dfd5c","Type":"ContainerStarted","Data":"1263f8d88e2e150ba5843871b613ff800d7de4b5bde5dadfe364588bece59e22"}
Nov 03 01:05:21.196768 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:21.196721    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" podStartSLOduration=1.7173391580000001 podStartE2EDuration="2.196707379s" totalImagesPullingTime="479.368221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:21.192088559 +0000 UTC m=+1466.405832232" watchObservedRunningTime="2026-11-03 01:05:21.196707379 +0000 UTC m=+1466.410451052"
Nov 03 01:05:24.197803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.197773    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerID="65abfb1c02afecb876507a843d04f8d4628e2b7ba00367b5129a3d21798725b1" exitCode=0
Nov 03 01:05:24.198134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.197842    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" event={"ID":"2d188bfe-c7cd-48f5-be7d-9416a81b417c","Type":"ContainerDied","Data":"65abfb1c02afecb876507a843d04f8d4628e2b7ba00367b5129a3d21798725b1"}
Nov 03 01:05:24.198134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.197893    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92" event={"ID":"2d188bfe-c7cd-48f5-be7d-9416a81b417c","Type":"ContainerDied","Data":"227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea"}
Nov 03 01:05:24.198134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.197911    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="227b643f85c6d88518b7b5751e51210fb97f1504a19cb4d3accc2b5c1d4471ea"
Nov 03 01:05:24.198276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.198224    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 01:05:24.231962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.231926    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232009    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232073    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232121    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232153    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232265    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232266    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:24.232335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232322    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232376    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.232553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232374    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:24.232553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:24.232553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\") pod \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\" (UID: \"2d188bfe-c7cd-48f5-be7d-9416a81b417c\") "
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.232937    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233095    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233237    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233253    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233263    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233272    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.233303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.233280    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.235075 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.235047    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:24.235410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.235383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "kube-api-access-46xf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:24.235837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.235808    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:24.236774 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.236752    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo" pod "2d188bfe-c7cd-48f5-be7d-9416a81b417c" (UID: "2d188bfe-c7cd-48f5-be7d-9416a81b417c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:05:24.334833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.334794    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.334833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.334826    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.334833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.334838    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d188bfe-c7cd-48f5-be7d-9416a81b417c-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:24.335075 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:24.334853    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xf6\" (UniqueName: \"kubernetes.io/projected/2d188bfe-c7cd-48f5-be7d-9416a81b417c-kube-api-access-46xf6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:05:25.205974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:25.205892    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"
Nov 03 01:05:25.227643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:25.227595    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 01:05:25.231660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:25.231639    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-v2j92"]
Nov 03 01:05:26.925185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:26.923274    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" path="/var/lib/kubelet/pods/2d188bfe-c7cd-48f5-be7d-9416a81b417c/volumes"
Nov 03 01:05:27.191040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:27.190510    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:05:31.002018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.001981    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:05:31.002693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.002675    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy"
Nov 03 01:05:31.002748 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.002696    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy"
Nov 03 01:05:31.002825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.002812    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d188bfe-c7cd-48f5-be7d-9416a81b417c" containerName="istio-proxy"
Nov 03 01:05:31.008013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.007990    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.016579 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.016558    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-4af1f7-serving-cert\""
Nov 03 01:05:31.017520 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.017496    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:05:31.143090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.143049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.143301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.143232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.143301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.143282    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.143422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.143302    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlw4\" (UniqueName: \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.244994 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.244953    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.245218 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.245126    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.245218 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.245192    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.245342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.245223    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlw4\" (UniqueName: \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.246340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.246301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.247636 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.247607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.247773 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.247757    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.253934 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.253862    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlw4\" (UniqueName: \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\") pod \"maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.352122 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.352089    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:31.643119 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:31.643079    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:05:31.643653 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:05:31.643623    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a603e3_e890_4171_a0e8_7afbd274bbc5.slice/crio-668a5c55271d4d1af31f274d912ea69fec91ab210f9827a888992a4574f6c9d8 WatchSource:0}: Error finding container 668a5c55271d4d1af31f274d912ea69fec91ab210f9827a888992a4574f6c9d8: Status 404 returned error can't find the container with id 668a5c55271d4d1af31f274d912ea69fec91ab210f9827a888992a4574f6c9d8
Nov 03 01:05:32.306506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:32.306466    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" event={"ID":"20a603e3-e890-4171-a0e8-7afbd274bbc5","Type":"ContainerStarted","Data":"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"}
Nov 03 01:05:32.306965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:32.306578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" event={"ID":"20a603e3-e890-4171-a0e8-7afbd274bbc5","Type":"ContainerStarted","Data":"668a5c55271d4d1af31f274d912ea69fec91ab210f9827a888992a4574f6c9d8"}
Nov 03 01:05:32.306965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:32.306609    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:32.358255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:32.357353    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" podStartSLOduration=1.90300929 podStartE2EDuration="2.357331444s" totalImagesPullingTime="454.322154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:32.323223652 +0000 UTC m=+1477.536967324" watchObservedRunningTime="2026-11-03 01:05:32.357331444 +0000 UTC m=+1477.571075117"
Nov 03 01:05:33.653644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.653598    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:05:33.658886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.658862    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.683040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.683008    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:05:33.802585 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802550    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.802585 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802595    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.802793 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802674    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zml7\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.802830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802786    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.802830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802821    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.802888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802858    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.803005 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.802978    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.803042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.803018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.803104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.803092    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904567    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904622    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zml7\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904708    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904748    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.904852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904793    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904874    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.904947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.905213    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.905304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905388 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.905373    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.905403    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.905497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.905478    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.907070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.907047    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.907259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.907243    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.915731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.915710    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.916041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.916017    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zml7\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\") pod \"e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") " pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:33.990463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:33.990432    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:34.296567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:34.296462    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:05:34.386637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:34.385064    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:34.386637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:34.385173    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:34.386637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:34.385216    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:34.431613 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:34.431575    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" event={"ID":"f0d1deb8-7cc5-4b91-a115-a9f21c460393","Type":"ContainerStarted","Data":"5ac6794663143b80050bbfa75ec1046fd23a3cae948b2f3fee0210c7fefe5451"}
Nov 03 01:05:35.459182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:35.454413    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" event={"ID":"f0d1deb8-7cc5-4b91-a115-a9f21c460393","Type":"ContainerStarted","Data":"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"}
Nov 03 01:05:35.482629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:35.482578    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podStartSLOduration=2.482563361 podStartE2EDuration="2.482563361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:35.48192415 +0000 UTC m=+1480.695667832" watchObservedRunningTime="2026-11-03 01:05:35.482563361 +0000 UTC m=+1480.696307032"
Nov 03 01:05:35.990937 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:35.990706    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:35.992148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:35.992118    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:35.992312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:35.992192    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:36.804772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.804687    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:05:36.809550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.809527    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:36.813067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.813042    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-4a697f-serving-cert\""
Nov 03 01:05:36.866293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.866256    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:05:36.908248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.908212    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:36.908597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.908576    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zl2\" (UniqueName: \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:36.908859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.908841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:36.908978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.908963    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:36.991275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.991238    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:36.991440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:36.991302    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:37.011140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.011088    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zl2\" (UniqueName: \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.011365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.011344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.011553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.011536    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.012456 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.012429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.012643 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:37.012627    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-serving-cert: secret "maas-api-e2e-shared-b-w4-4a697f-serving-cert" not found
Nov 03 01:05:37.012737 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:37.012685    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls podName:3865d725-5e1d-4dd6-ab4c-071637eadfad nodeName:}" failed. No retries permitted until 2026-11-03 01:05:37.512667932 +0000 UTC m=+1482.726411601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls") pod "maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" (UID: "3865d725-5e1d-4dd6-ab4c-071637eadfad") : secret "maas-api-e2e-shared-b-w4-4a697f-serving-cert" not found
Nov 03 01:05:37.012737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.012324    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.014757 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.014708    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.024318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.024289    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zl2\" (UniqueName: \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.522462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.522414    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.525428 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.525400    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.770105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.762462    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:37.991179 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.990786    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:37.991179 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:37.990836    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:38.263259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.263114    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:05:38.321221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.321126    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:05:38.555226 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.553533    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" event={"ID":"3865d725-5e1d-4dd6-ab4c-071637eadfad","Type":"ContainerStarted","Data":"4e6bf70a165b3d9e84d47fe794564e761ab2a925ce99ecc1dc6b56b10affa2af"}
Nov 03 01:05:38.761206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.751400    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:05:38.787369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.779372    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:05:38.787369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.779465    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:38.796001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.795644    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-50e623-serving-cert\""
Nov 03 01:05:38.918066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.908086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7bw\" (UniqueName: \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:38.918066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.908208    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:38.918066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.908272    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:38.918066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.914570    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:38.994734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.994265    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:38.994734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:38.994327    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:39.015880 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.015842    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7bw\" (UniqueName: \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.016061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.015931    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.016061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.015987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.016210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.016095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.017007 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:39.016432    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-serving-cert: secret "maas-api-e2e-shared-a-w0-50e623-serving-cert" not found
Nov 03 01:05:39.017007 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:39.016505    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls podName:d03d2a92-b898-466b-a679-b94d45070b0d nodeName:}" failed. No retries permitted until 2026-11-03 01:05:39.516483351 +0000 UTC m=+1484.730227021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls") pod "maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" (UID: "d03d2a92-b898-466b-a679-b94d45070b0d") : secret "maas-api-e2e-shared-a-w0-50e623-serving-cert" not found
Nov 03 01:05:39.017007 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.016794    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.019068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.019045    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.033346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.033314    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7bw\" (UniqueName: \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.537184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.536279    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.558206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.558174    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.584572 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.584521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" event={"ID":"3865d725-5e1d-4dd6-ab4c-071637eadfad","Type":"ContainerStarted","Data":"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"}
Nov 03 01:05:39.609870 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.609810    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" podStartSLOduration=3.204297973 podStartE2EDuration="3.609791626s" totalImagesPullingTime="405.493653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:39.601807059 +0000 UTC m=+1484.815550731" watchObservedRunningTime="2026-11-03 01:05:39.609791626 +0000 UTC m=+1484.823535299"
Nov 03 01:05:39.708180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.702289    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:39.993185 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:39.992890    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:05:39.993340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.993308    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:39.993433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:39.993357    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:40.366924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:40.366144    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:05:40.595757 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:40.595724    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" event={"ID":"d03d2a92-b898-466b-a679-b94d45070b0d","Type":"ContainerStarted","Data":"c3ccfc368fd0b9471c2c35ca921cfd5c95c31b51ecaa04725e053a4a787071c7"}
Nov 03 01:05:40.595956 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:40.595808    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:40.992666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:40.992625    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:05:40.992831 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:40.992691    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:05:41.605250 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:41.605214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" event={"ID":"d03d2a92-b898-466b-a679-b94d45070b0d","Type":"ContainerStarted","Data":"38e810e58be171db4108c8285d688d1a1d82ebb17b0c84ad3005524f9cf43871"}
Nov 03 01:05:41.605784 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:41.605633    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:41.629008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:41.628938    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" podStartSLOduration=3.198085514 podStartE2EDuration="3.628920248s" totalImagesPullingTime="430.834734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:41.623703996 +0000 UTC m=+1486.837447668" watchObservedRunningTime="2026-11-03 01:05:41.628920248 +0000 UTC m=+1486.842663923"
Nov 03 01:05:42.371426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:42.371384    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:42.371743 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:42.371718    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:42.372707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:42.372684    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:05:45.094127 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.094082    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:05:45.101063 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.099429    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.106007 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.105982    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2729f2-openshift-default-dockercfg-9nk2m\""
Nov 03 01:05:45.114999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.114965    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:05:45.205527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205492    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205587    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205729    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5j6\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205754    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.205923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.205908    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.206089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.206068    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.206247 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.206228    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308132    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308314    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308355    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308403    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308444    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5j6\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308533 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308533 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308515    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308639 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.308987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308960    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.309082 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.308961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.309082 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.309022    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.311192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.311133    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.311481 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.311459    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.316207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.316134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.316423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.316407    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5j6\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.443557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.443471    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:45.749804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.749756    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:45.749930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.749875    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:45.749930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.749918    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:05:45.751595 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:45.751567    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:05:46.611463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:46.611436    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:05:46.667195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:46.662450    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" event={"ID":"f6d1eaab-3098-4887-ac53-752148bdcfec","Type":"ContainerStarted","Data":"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"}
Nov 03 01:05:46.667195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:46.662521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" event={"ID":"f6d1eaab-3098-4887-ac53-752148bdcfec","Type":"ContainerStarted","Data":"0c71cce9b9c64b123428b364ff3b7956b55b3bbb3f0676db183d2db1b71f8ccd"}
Nov 03 01:05:46.689229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:46.689007    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podStartSLOduration=1.68899026 podStartE2EDuration="1.68899026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:46.684609746 +0000 UTC m=+1491.898353418" watchObservedRunningTime="2026-11-03 01:05:46.68899026 +0000 UTC m=+1491.902733935"
Nov 03 01:05:47.458261 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:47.449501    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:47.458642 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:47.458609    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 01:05:47.459196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:47.459134    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 01:05:47.617231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:47.617197    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:05:48.449186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:48.445686    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 01:05:48.449186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:48.445764    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 01:05:49.449595 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:49.449552    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 01:05:49.450232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:49.450199    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 01:05:50.449202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:50.448591    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 01:05:50.449202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:50.448652    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 01:05:51.447676 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:51.447639    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 01:05:51.448318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:51.448268    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 01:05:52.450855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:52.450817    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:52.451336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:52.451281    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:52.452103 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:52.452077    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:05:54.162258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.162204    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:05:54.180401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.180358    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.181816 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.181791    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:05:54.184840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.184817    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-556c14-serving-cert\""
Nov 03 01:05:54.283741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.283702    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlprx\" (UniqueName: \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.283912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.283821    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.283985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.283968    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.284109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.284096    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.385300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.385266    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.385503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.385338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlprx\" (UniqueName: \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.385503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.385401    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.385633 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.385504    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.385633 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:54.385538    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-serving-cert: secret "maas-api-e2e-shared-b-w3-556c14-serving-cert" not found
Nov 03 01:05:54.385633 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:05:54.385615    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls podName:00535132-f427-428e-9b3d-301eeace95db nodeName:}" failed. No retries permitted until 2026-11-03 01:05:54.885594102 +0000 UTC m=+1500.099337768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls") pod "maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" (UID: "00535132-f427-428e-9b3d-301eeace95db") : secret "maas-api-e2e-shared-b-w3-556c14-serving-cert" not found
Nov 03 01:05:54.386019 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.385994    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.388021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.388000    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.396804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.396783    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlprx\" (UniqueName: \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.894924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.894888    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:54.897962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:54.897933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:55.101957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:55.101920    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:55.441066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:55.440984    4531 scope.go:122] "RemoveContainer" containerID="65abfb1c02afecb876507a843d04f8d4628e2b7ba00367b5129a3d21798725b1"
Nov 03 01:05:55.494494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:55.494457    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:05:55.510345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:55.510314    4531 scope.go:122] "RemoveContainer" containerID="d732ca02e131741a7b44c99779037045ceb6ec6ac56622e0e763df1a87c85945"
Nov 03 01:05:55.895648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:55.895611    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" event={"ID":"00535132-f427-428e-9b3d-301eeace95db","Type":"ContainerStarted","Data":"9c683f7be6fcdc498f1e99b4d814a816896a4ab8f9c5755e32bff7f943bb8251"}
Nov 03 01:05:56.904924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:56.904877    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" event={"ID":"00535132-f427-428e-9b3d-301eeace95db","Type":"ContainerStarted","Data":"2dfa38a7dc220be4bcba5bd8c02343acdde8327e041bba04861d83b775560eea"}
Nov 03 01:05:56.905401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:56.904949    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:05:56.926622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:05:56.926575    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" podStartSLOduration=2.48327669 podStartE2EDuration="2.926560444s" totalImagesPullingTime="443.283754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:05:56.923984 +0000 UTC m=+1502.137727673" watchObservedRunningTime="2026-11-03 01:05:56.926560444 +0000 UTC m=+1502.140304116"
Nov 03 01:06:02.917963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:02.917932    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:06:10.010883 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:06:10.010838    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:06:15.185103 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.185067    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:06:15.189655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.189633    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.204402 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.204372    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:06:15.287588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.287545    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:06:15.291906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.291879    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-717217-openshift-default-dockercfg-4kr2v\""
Nov 03 01:06:15.292432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.292409    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.305821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.305795    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:06:15.313394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrl6m\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313531 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313403    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313565    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313645 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.313923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313877    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.314015 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.313999    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.314129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.314114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.393185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.392282    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-4af1f7-openshift-default-dockercfg-dmrp9\""
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419179    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419302    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419559    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419629    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhcb\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419683    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.420231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.419733    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.423040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.423590    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424364    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424432    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424664    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424746    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrl6m\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.424784    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.425148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.445231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.426208    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.470183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.447010    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.470183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.450721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrl6m\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.470183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.455753    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.486306 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.479703    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.528221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.527440    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528583    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528688    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhcb\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528741    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528800    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.528930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.529024    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.529078    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.529654    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.529915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.530128    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.530438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.532183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.530547    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.559201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.554124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.559201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.555132    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.559201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.557045    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.559201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.558376    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhcb\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.613840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.613804    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:15.922734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.922688    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:15.922918 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.922859    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:15.923035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.922973    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:15.927020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.926938    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:06:15.946628 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:06:15.946595    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762bfce8_cd1a_4dd3_ada4_64084f8d2eea.slice/crio-821d2ca5bec2d923f379412faf18afd5766d4fb57741716b9b1a3e30023a2781 WatchSource:0}: Error finding container 821d2ca5bec2d923f379412faf18afd5766d4fb57741716b9b1a3e30023a2781: Status 404 returned error can't find the container with id 821d2ca5bec2d923f379412faf18afd5766d4fb57741716b9b1a3e30023a2781
Nov 03 01:06:15.948560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.948434    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:06:15.949626 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.949592    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:15.949713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.949669    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:15.949754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:15.949717    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:16.127305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.127265    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" event={"ID":"762bfce8-cd1a-4dd3-ada4-64084f8d2eea","Type":"ContainerStarted","Data":"92cdf306309dd596140acba9d23a13b803021a1ed221e60b3f31f349e634ad03"}
Nov 03 01:06:16.127477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.127354    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" event={"ID":"762bfce8-cd1a-4dd3-ada4-64084f8d2eea","Type":"ContainerStarted","Data":"821d2ca5bec2d923f379412faf18afd5766d4fb57741716b9b1a3e30023a2781"}
Nov 03 01:06:16.130755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.130725    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" event={"ID":"ba6f631a-2510-4c8e-9f9d-791087aa43e1","Type":"ContainerStarted","Data":"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"}
Nov 03 01:06:16.130864 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.130788    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" event={"ID":"ba6f631a-2510-4c8e-9f9d-791087aa43e1","Type":"ContainerStarted","Data":"8a96e9d841e7d15d8ea15feb1497dd234ac12627bed5530bfd6bfe377a82b875"}
Nov 03 01:06:16.180254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.179399    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podStartSLOduration=1.179383414 podStartE2EDuration="1.179383414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:16.173839785 +0000 UTC m=+1521.387583458" watchObservedRunningTime="2026-11-03 01:06:16.179383414 +0000 UTC m=+1521.393127085"
Nov 03 01:06:16.540185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.534594    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:16.619193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:16.618253    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:17.677196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:17.669543    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:17.677196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:17.669599    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:17.702193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:17.701321    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:17.702193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:17.701384    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:18.489618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.489551    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podStartSLOduration=3.4895279869999998 podStartE2EDuration="3.489527987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:16.229310911 +0000 UTC m=+1521.443054585" watchObservedRunningTime="2026-11-03 01:06:18.489527987 +0000 UTC m=+1523.703271663"
Nov 03 01:06:18.502863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.502829    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:06:18.526333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.526286    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:06:18.526729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.526654    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.530764 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.530577    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:18.540405 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.540354    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:18.616284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.616250    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:18.616487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.616461    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:18.622474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.622436    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-50e623-openshift-default-dockercfg-ngqv2\""
Nov 03 01:06:18.807237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.786338    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.806843    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.806977    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807045    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807260    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807539    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807572    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmzb\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.807908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.807608    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.909732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.909441    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.909973 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.909942    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910630 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910602    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910778 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910762    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910808    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmzb\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910854    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910917    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.910967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.910943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.911081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.911038    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.911523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.911500    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.911758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.911737    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.912895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.912837    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.913491 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.913471    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.916587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.916259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.918147 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.917810    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.918147 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.918108    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.920213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.920190    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmzb\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:18.958186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:18.926156    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:19.401262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.400330    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:06:19.410270 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:06:19.410237    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70ae258_f4f8_41a2_988b_74f9e7d75fc3.slice/crio-ddd75ed7d7d369bf5e37d7e693c2ef3edd940799cf4a4d8adf8df3be0c308fa7 WatchSource:0}: Error finding container ddd75ed7d7d369bf5e37d7e693c2ef3edd940799cf4a4d8adf8df3be0c308fa7: Status 404 returned error can't find the container with id ddd75ed7d7d369bf5e37d7e693c2ef3edd940799cf4a4d8adf8df3be0c308fa7
Nov 03 01:06:19.413481 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.413404    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:19.413481 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.413472    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:19.413596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.413505    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:19.541422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.536606    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:19.541422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.536657    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:19.614519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.614493    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:19.614657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:19.614536    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:20.318499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.318461    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" event={"ID":"e70ae258-f4f8-41a2-988b-74f9e7d75fc3","Type":"ContainerStarted","Data":"262b842b3a5d644c37e0197450cad6c918999286bec13131f1cc7eed52bc1aee"}
Nov 03 01:06:20.319013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.318538    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" event={"ID":"e70ae258-f4f8-41a2-988b-74f9e7d75fc3","Type":"ContainerStarted","Data":"ddd75ed7d7d369bf5e37d7e693c2ef3edd940799cf4a4d8adf8df3be0c308fa7"}
Nov 03 01:06:20.531186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.530535    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:20.531186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.530596    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:20.615415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.615371    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:20.615644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.615434    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:20.945246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.942886    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:20.945246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.942948    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:20.972369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:20.972335    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:21.542186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.531604    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:21.542186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.531685    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:21.621185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.617103    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:21.621185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.617182    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:22.066189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.989280    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:22.066189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:21.989345    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:22.163243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.156569    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podStartSLOduration=4.15653471 podStartE2EDuration="4.15653471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:20.382111627 +0000 UTC m=+1525.595855298" watchObservedRunningTime="2026-11-03 01:06:22.15653471 +0000 UTC m=+1527.370278383"
Nov 03 01:06:22.236301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.236262    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:06:22.450889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.450847    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:06:22.497635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.454306    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.497635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.489778    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-020af9-serving-cert\""
Nov 03 01:06:22.497635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.490705    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.497635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.490763    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.595187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.585408    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:06:22.595187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.585466    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:06:22.611192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.607477    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.611192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.607756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw65f\" (UniqueName: \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.611192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.607825    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.611192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.607865    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.611192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.609857    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.615700 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.615672    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Nov 03 01:06:22.616650 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.615913    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Nov 03 01:06:22.644226 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.627841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.715292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.714858    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.715292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.715070    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw65f\" (UniqueName: \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.773185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.769946    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw65f\" (UniqueName: \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.776701 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.772938    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.918143 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.918110    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:22.936448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.936410    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:22.936673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:22.936641    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:23.470212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.465023    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:06:23.546186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.543660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" event={"ID":"d3e0b936-03da-4006-9d61-75118e093a9d","Type":"ContainerStarted","Data":"e1bc4cf17d9b84fe287463fea7d8cd82b960ea29c7ea46e8b1603bbdd453a0e3"}
Nov 03 01:06:23.752242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.733523    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:23.752242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.735046    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:23.797236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.767790    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:06:23.924969 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.924841    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:23.930327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.926407    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:23.982482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.972342    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:23.982482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.972413    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:23.996124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:23.995943    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:06:24.953476 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:24.953431    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:24.954087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:24.954052    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:25.586302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:25.586261    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" event={"ID":"d3e0b936-03da-4006-9d61-75118e093a9d","Type":"ContainerStarted","Data":"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"}
Nov 03 01:06:25.586480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:25.586418    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:25.608759 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:25.608707    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" podStartSLOduration=3.846411551 podStartE2EDuration="4.608693319s" totalImagesPullingTime="762.281768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:25.605926441 +0000 UTC m=+1530.819670113" watchObservedRunningTime="2026-11-03 01:06:25.608693319 +0000 UTC m=+1530.822436990"
Nov 03 01:06:25.930647 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:25.930549    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:25.930918 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:25.930888    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:26.972183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:26.941790    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 01:06:26.972183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:26.941851    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 01:06:27.934932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:27.934894    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:27.935302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:27.935270    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:27.935989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:27.935963    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:06:30.129574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.129543    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:06:30.135975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.135945    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.160859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.160824    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:06:30.231780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.231733    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.231962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.231791    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.231962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.231934    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.231974    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232128 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.232102    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.232253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.232285    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.232417    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nzc\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.232507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.232493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334357    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nzc\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334451    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334647    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334686    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334736    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334818    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.334930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.334847    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.335112 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.335085    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.335361 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.335334    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.335466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.335403    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.335466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.335450    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.335554 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.335535    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.337120 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.337094    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.338013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.337993    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.342398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.342373    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.342678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.342658    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nzc\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\") pod \"e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") " pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.455325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.455235    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:30.755741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.755712    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:06:30.756195 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:06:30.756135    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c2c1d6_30ac_4b37_993d_e667f98d6e43.slice/crio-5c41e3ef47491eefe1d044c39a7a563fade87632a41400a1779a71ae4e02882e WatchSource:0}: Error finding container 5c41e3ef47491eefe1d044c39a7a563fade87632a41400a1779a71ae4e02882e: Status 404 returned error can't find the container with id 5c41e3ef47491eefe1d044c39a7a563fade87632a41400a1779a71ae4e02882e
Nov 03 01:06:30.759487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.759453    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:30.759568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.759522    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:30.759568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:30.759551    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:06:31.601476 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:31.601445    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:06:31.673983 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:31.673411    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" event={"ID":"e5c2c1d6-30ac-4b37-993d-e667f98d6e43","Type":"ContainerStarted","Data":"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"}
Nov 03 01:06:31.673983 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:31.673478    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" event={"ID":"e5c2c1d6-30ac-4b37-993d-e667f98d6e43","Type":"ContainerStarted","Data":"5c41e3ef47491eefe1d044c39a7a563fade87632a41400a1779a71ae4e02882e"}
Nov 03 01:06:32.455579 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:32.455546    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:32.457496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:32.457123    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 01:06:32.457496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:32.457204    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 01:06:33.456754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:33.456712    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 01:06:33.457419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:33.457368    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 01:06:34.457191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:34.456534    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 01:06:34.457191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:34.456648    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 01:06:35.457184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:35.456269    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 01:06:35.457184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:35.456331    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 01:06:36.459327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:36.456539    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 01:06:36.459327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:36.456602    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 01:06:37.461780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:37.461729    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:37.462443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:37.462415    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:37.463305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:37.463282    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:06:37.484053 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:06:37.484010    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podStartSLOduration=7.483997612 podStartE2EDuration="7.483997612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:31.700447646 +0000 UTC m=+1536.914191311" watchObservedRunningTime="2026-11-03 01:06:37.483997612 +0000 UTC m=+1542.697741283"
Nov 03 01:06:39.878723 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:06:39.878686    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:07:00.268459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.268423    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:07:00.275405 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.275377    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.311334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.311298    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:07:00.359248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.359188    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:07:00.364719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.364692    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.394812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.394812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394666    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.394812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394741    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394838    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394883    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394919    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.394997    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.395041    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.395121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.395076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96m2b\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.399106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.398800    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:07:00.496413 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496376    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.496627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496433    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.496627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496509    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.496627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496555    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96m2b\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.496627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496612    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.496866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496636    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.496866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496683    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.496866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496757    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.497034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496882    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496899    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.497034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496942    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.497034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.496990    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497052    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp48f\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.497294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497118    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497202    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497241    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497607    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497659    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497684    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497699    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.497910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497753    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.497910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.497880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.498142 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.498117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.499671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.499349    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.500431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.500410    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.505782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.505763    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.506250 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.506232    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96m2b\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\") pod \"e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") " pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.599709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599713    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599793    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599813    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599859    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599885    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599928    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.599988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.599945    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp48f\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600227    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600457 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600435    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600534    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.600750 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.600721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.602291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.602270    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.602579 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.602559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.606489 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.606466    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:00.611717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.611698    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.612822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.612800    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp48f\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\") pod \"e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") " pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:00.698249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:00.697882    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:01.048605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.031108    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:07:01.048605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.038657    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:07:01.072735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.072689    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.073030 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.073004    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.073253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.073232    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.079993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.079961    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.093281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.093246    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.093529 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.093507    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:01.115223 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.115184    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" event={"ID":"be64d1aa-dda7-4951-988c-abdf2854a300","Type":"ContainerStarted","Data":"24d00742c051adcc243d3c2e9c75b86505cd4245c70250a77b49059a483df5fa"}
Nov 03 01:07:01.117153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:01.117120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" event={"ID":"5940c844-7285-498e-8bae-47ecfe2eb48d","Type":"ContainerStarted","Data":"da8093a2518967937003785dcd781208da35bf8c186612f9d01c6849c6d61084"}
Nov 03 01:07:02.145101 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.145051    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" event={"ID":"be64d1aa-dda7-4951-988c-abdf2854a300","Type":"ContainerStarted","Data":"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"}
Nov 03 01:07:02.150634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.149876    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" event={"ID":"5940c844-7285-498e-8bae-47ecfe2eb48d","Type":"ContainerStarted","Data":"b09fdcbd8568004d0b6ce438e7f1f39229ae45f1b77f01837050ad91bbd3dae4"}
Nov 03 01:07:02.325385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.325316    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podStartSLOduration=2.3253009860000002 podStartE2EDuration="2.325300986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:02.191680683 +0000 UTC m=+1567.405424371" watchObservedRunningTime="2026-11-03 01:07:02.325300986 +0000 UTC m=+1567.539044741"
Nov 03 01:07:02.348208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.334766    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podStartSLOduration=2.334746142 podStartE2EDuration="2.334746142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:02.302003081 +0000 UTC m=+1567.515746753" watchObservedRunningTime="2026-11-03 01:07:02.334746142 +0000 UTC m=+1567.548489815"
Nov 03 01:07:02.612022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.610523    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:02.626187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.621693    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:02.626187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.621755    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:02.707181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.704569    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:02.707181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.704898    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:02.707181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:02.704952    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:03.325705 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.292148    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:07:03.326597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.326560    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" podUID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" containerName="maas-api" containerID="cri-o://b0d7c239f3a76baaa820afca69fcb858b95d957ddf99ca9acbe454f4cdf59c77" gracePeriod=30
Nov 03 01:07:03.611057 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.610557    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:03.611057 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.610621    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:03.713186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.698631    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:03.713186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.698695    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:03.813953 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.738416    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:07:03.813953 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.744448    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:03.813953 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:03.792263    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.163535    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164050    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164181    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164378    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164407    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164437    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164478    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvpq\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.168183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.164547    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.185725 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.185694    4531 generic.go:356] "Generic (PLEG): container finished" podID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" containerID="b0d7c239f3a76baaa820afca69fcb858b95d957ddf99ca9acbe454f4cdf59c77" exitCode=0
Nov 03 01:07:04.186912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.186884    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" event={"ID":"4fb51cb4-357a-45ef-9983-61c3125dfd5c","Type":"ContainerDied","Data":"b0d7c239f3a76baaa820afca69fcb858b95d957ddf99ca9acbe454f4cdf59c77"}
Nov 03 01:07:04.298234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.284785    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.311309 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.289739    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.331101 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.331061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.332213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.331915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.332553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.332529    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.333821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.333801    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.334563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.334544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.334984 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.334967    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.335770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.335753    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvpq\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.336242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.336224    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.379905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.371413    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.399226 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.334919    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.399226 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.336738    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.399226 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.335686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.408811 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.406096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvpq\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.424140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.424103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.478129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.459470    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.526478 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.525760    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:04.561526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.549475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:04.595631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.595585    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:04.623564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.623004    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:04.623564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.623062    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:04.796115 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.796079    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:04.796462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.796428    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:04.797398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:04.797376    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:05.129626 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.129321    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:07:05.216870 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.216787    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"
Nov 03 01:07:05.217027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.216791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94" event={"ID":"4fb51cb4-357a-45ef-9983-61c3125dfd5c","Type":"ContainerDied","Data":"1263f8d88e2e150ba5843871b613ff800d7de4b5bde5dadfe364588bece59e22"}
Nov 03 01:07:05.217132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.217121    4531 scope.go:122] "RemoveContainer" containerID="b0d7c239f3a76baaa820afca69fcb858b95d957ddf99ca9acbe454f4cdf59c77"
Nov 03 01:07:05.369184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.368790    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\" (UniqueName: \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\") pod \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") "
Nov 03 01:07:05.369184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.368980    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\") pod \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") "
Nov 03 01:07:05.369184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.369131    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\") pod \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") "
Nov 03 01:07:05.369759 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.369234    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\") pod \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\" (UID: \"4fb51cb4-357a-45ef-9983-61c3125dfd5c\") "
Nov 03 01:07:05.392692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.392653    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca" pod "4fb51cb4-357a-45ef-9983-61c3125dfd5c" (UID: "4fb51cb4-357a-45ef-9983-61c3125dfd5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:05.395137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.395080    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls" pod "4fb51cb4-357a-45ef-9983-61c3125dfd5c" (UID: "4fb51cb4-357a-45ef-9983-61c3125dfd5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:05.396621 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.396592    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk" pod "4fb51cb4-357a-45ef-9983-61c3125dfd5c" (UID: "4fb51cb4-357a-45ef-9983-61c3125dfd5c"). InnerVolumeSpecName "kube-api-access-7lzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:05.397751 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.397728    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs" pod "4fb51cb4-357a-45ef-9983-61c3125dfd5c" (UID: "4fb51cb4-357a-45ef-9983-61c3125dfd5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:05.499283 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.488195    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzgk\" (UniqueName: \"kubernetes.io/projected/4fb51cb4-357a-45ef-9983-61c3125dfd5c-kube-api-access-7lzgk\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:07:05.499283 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.488246    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:07:05.499283 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.488265    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb51cb4-357a-45ef-9983-61c3125dfd5c-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:07:05.499283 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.488280    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fb51cb4-357a-45ef-9983-61c3125dfd5c-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:07:05.506429 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.506387    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:07:05.519067 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:07:05.519032    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9545e67c_f608_4686_be71_89cc29823e3e.slice/crio-487517a1f55d1cc8d7c70e16c61e3139e35a9ab04f5392527b93c29e18b19f71 WatchSource:0}: Error finding container 487517a1f55d1cc8d7c70e16c61e3139e35a9ab04f5392527b93c29e18b19f71: Status 404 returned error can't find the container with id 487517a1f55d1cc8d7c70e16c61e3139e35a9ab04f5392527b93c29e18b19f71
Nov 03 01:07:05.524296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.523264    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:05.524296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.523371    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:05.524296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.523434    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:05.607780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.607743    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:05.607944 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.607804    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:05.620734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.620707    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:07:05.624895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.624867    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-4248ae-77dddff8b6-8fs94"]
Nov 03 01:07:05.700548 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.700512    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:05.700824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:05.700792    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:06.259230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.259188    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" event={"ID":"9545e67c-f608-4686-be71-89cc29823e3e","Type":"ContainerStarted","Data":"4cafa7f0b8266d7c4433683e6371fe8fc1bad75ebc9a9e097b703d7f19df3066"}
Nov 03 01:07:06.259519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.259487    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" event={"ID":"9545e67c-f608-4686-be71-89cc29823e3e","Type":"ContainerStarted","Data":"487517a1f55d1cc8d7c70e16c61e3139e35a9ab04f5392527b93c29e18b19f71"}
Nov 03 01:07:06.328055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.327981    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podStartSLOduration=3.327957183 podStartE2EDuration="3.327957183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:06.300338676 +0000 UTC m=+1571.514082346" watchObservedRunningTime="2026-11-03 01:07:06.327957183 +0000 UTC m=+1571.541700858"
Nov 03 01:07:06.610844 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.610754    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:06.611401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.611368    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:06.701184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.700512    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:06.701184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.700573    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:06.812181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.812023    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:06.812381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.812231    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:06.812381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.812278    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:06.989239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:06.984561    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" path="/var/lib/kubelet/pods/4fb51cb4-357a-45ef-9983-61c3125dfd5c/volumes"
Nov 03 01:07:07.609186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.608954    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:07.609186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.609017    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:07.705019 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.700452    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:07.705019 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.700508    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:07.853187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.848328    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:07.853187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:07.848389    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:08.466188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.464819    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:08.591184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.587249    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:08.625188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.618144    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 01:07:08.625188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.618367    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 01:07:08.698315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.698259    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:08.700190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.699676    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:07:08.700190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.699721    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:07:08.798599 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.798504    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:08.799154 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:08.799126    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:09.755185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:09.754494    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:09.759182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:09.756226    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:09.768221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:09.767659    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:07:09.804888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:09.804323    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:09.804888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:09.804394    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:09.990624 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:07:09.990585    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:07:10.016778 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:10.016629    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:10.018644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:10.018621    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:10.019838 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:10.019770    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:07:10.798672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:10.798634    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:10.798844 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:10.798687    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:11.798225 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:11.798152    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:11.798592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:11.798250    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:12.798597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:12.798564    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:12.799064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:12.798637    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:13.798333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:13.798297    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 01:07:13.798508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:13.798360    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 01:07:14.803296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:14.803262    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:14.803687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:14.803665    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:14.804431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:14.804412    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:07:18.563232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.563200    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:07:18.564194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.564147    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" containerName="maas-api"
Nov 03 01:07:18.564194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.564185    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" containerName="maas-api"
Nov 03 01:07:18.564342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.564332    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb51cb4-357a-45ef-9983-61c3125dfd5c" containerName="maas-api"
Nov 03 01:07:18.568258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.568233    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.583997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.583972    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:07:18.664258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.664221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.664644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.664610    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.664810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.664788    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.664925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.664862    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.664990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.664940    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.665046 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.665006    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.665101 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.665062    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.665156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.665114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.665250 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.665196    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wn7\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.767585 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767549    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.767781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.767781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767676    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.767781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767734    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.767951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767784    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wn7\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768015 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.767981    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768054    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768200 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768147    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768194    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768196    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768377    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.768533 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.768499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.770485 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.770452    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.770826 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.770799    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.782422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.782393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.782702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.782675    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wn7\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\") pod \"e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") " pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:18.887323 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:18.887289    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:19.191561 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:07:19.191529    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf9ec75_9c0b_4c2e_a85b_4901e3ffd4bf.slice/crio-cb0b5ce73a254b5b067605d105e2468ba855344d7bab35c5f85d0c4e2e59d813 WatchSource:0}: Error finding container cb0b5ce73a254b5b067605d105e2468ba855344d7bab35c5f85d0c4e2e59d813: Status 404 returned error can't find the container with id cb0b5ce73a254b5b067605d105e2468ba855344d7bab35c5f85d0c4e2e59d813
Nov 03 01:07:19.191944 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.191916    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:07:19.194530 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.194498    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:19.194648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.194566    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:19.194648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.194595    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:07:19.535837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.534787    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" event={"ID":"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf","Type":"ContainerStarted","Data":"fb1d79abe62b243d142f19787536bda2f0057b09ff3ec6e4540f2518b2e07071"}
Nov 03 01:07:19.535837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.534854    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" event={"ID":"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf","Type":"ContainerStarted","Data":"cb0b5ce73a254b5b067605d105e2468ba855344d7bab35c5f85d0c4e2e59d813"}
Nov 03 01:07:19.563010 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.562517    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podStartSLOduration=1.562497939 podStartE2EDuration="1.562497939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:19.556491636 +0000 UTC m=+1584.770235310" watchObservedRunningTime="2026-11-03 01:07:19.562497939 +0000 UTC m=+1584.776241612"
Nov 03 01:07:19.887838 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:19.887802    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:20.898181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:20.896458    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:20.898181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:20.896530    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:21.888357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:21.888320    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:21.888541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:21.888373    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:22.887938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:22.887899    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:22.888375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:22.887956    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:23.612469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.612436    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:07:23.617511 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.617484    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.625469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.625437    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-84e377-serving-cert\""
Nov 03 01:07:23.675183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.672110    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:07:23.699835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.699801    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.699990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.699917    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sts67\" (UniqueName: \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.700056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.700015    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.700260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.700231    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.853188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.849504    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.853188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.849649    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sts67\" (UniqueName: \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.853188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.849769    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.853188 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:07:23.852774    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-serving-cert: secret "maas-api-e2e-explicit-84e377-serving-cert" not found
Nov 03 01:07:23.853188 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:07:23.852844    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls podName:cd89036a-534f-479a-b030-6ee35f289901 nodeName:}" failed. No retries permitted until 2026-11-03 01:07:24.352824678 +0000 UTC m=+1589.566568344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls") pod "maas-api-e2e-explicit-84e377-f8768b479-st7vl" (UID: "cd89036a-534f-479a-b030-6ee35f289901") : secret "maas-api-e2e-explicit-84e377-serving-cert" not found
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.849865    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.903805    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.903862    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.903983    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.904055    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:23.917249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:23.904398    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sts67\" (UniqueName: \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:24.417182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:24.416730    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:24.439184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:24.425065    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") pod \"maas-api-e2e-explicit-84e377-f8768b479-st7vl\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:24.607195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:24.606023    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:24.892339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:24.890575    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:24.892339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:24.890650    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:25.116604 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:07:25.116571    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd89036a_534f_479a_b030_6ee35f289901.slice/crio-efa97c6c6edcdbb0fb6c9de5947df5aa50271d4b1687308aa21025617598dd72 WatchSource:0}: Error finding container efa97c6c6edcdbb0fb6c9de5947df5aa50271d4b1687308aa21025617598dd72: Status 404 returned error can't find the container with id efa97c6c6edcdbb0fb6c9de5947df5aa50271d4b1687308aa21025617598dd72
Nov 03 01:07:25.118490 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:25.118462    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:07:25.666397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:25.666351    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" event={"ID":"cd89036a-534f-479a-b030-6ee35f289901","Type":"ContainerStarted","Data":"efa97c6c6edcdbb0fb6c9de5947df5aa50271d4b1687308aa21025617598dd72"}
Nov 03 01:07:25.888541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:25.888503    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 03 01:07:25.888755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:25.888558    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 03 01:07:26.676872 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.676828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" event={"ID":"cd89036a-534f-479a-b030-6ee35f289901","Type":"ContainerStarted","Data":"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"}
Nov 03 01:07:26.677277 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.676916    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:26.696945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.696887    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" podStartSLOduration=3.138917728 podStartE2EDuration="3.696869246s" totalImagesPullingTime="557.951518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:26.693584172 +0000 UTC m=+1591.907327846" watchObservedRunningTime="2026-11-03 01:07:26.696869246 +0000 UTC m=+1591.910612920"
Nov 03 01:07:26.892786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.892753    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:26.893100 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.893069    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:26.894029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:26.894010    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:07:32.690596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:07:32.690566    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:07:39.902770 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:07:39.902648    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:08:09.880100 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:08:09.880058    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:08:40.131255 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:08:40.127414    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:08:41.109393 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:41.108255    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:08:41.460397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:41.460283    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:08:44.016212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.016133    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:08:44.016693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.016462    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" podUID="cd89036a-534f-479a-b030-6ee35f289901" containerName="maas-api" containerID="cri-o://f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c" gracePeriod=30
Nov 03 01:08:44.770243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.770146    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:08:44.807447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.807418    4531 generic.go:356] "Generic (PLEG): container finished" podID="cd89036a-534f-479a-b030-6ee35f289901" containerID="f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c" exitCode=0
Nov 03 01:08:44.807627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.807455    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" event={"ID":"cd89036a-534f-479a-b030-6ee35f289901","Type":"ContainerDied","Data":"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"}
Nov 03 01:08:44.807627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.807489    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"
Nov 03 01:08:44.807627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.807527    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl" event={"ID":"cd89036a-534f-479a-b030-6ee35f289901","Type":"ContainerDied","Data":"efa97c6c6edcdbb0fb6c9de5947df5aa50271d4b1687308aa21025617598dd72"}
Nov 03 01:08:44.807627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.807557    4531 scope.go:122] "RemoveContainer" containerID="f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"
Nov 03 01:08:44.822328 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.822308    4531 scope.go:122] "RemoveContainer" containerID="f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"
Nov 03 01:08:44.822668 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:08:44.822644    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c\": container with ID starting with f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c not found: ID does not exist" containerID="f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"
Nov 03 01:08:44.822737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.822682    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c"} err="failed to get container status \"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c\": rpc error: code = NotFound desc = could not find container \"f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c\": container with ID starting with f26ec61051f46e95ce212e669366f85498d3b102606b3b43f66ebe5d3df7a06c not found: ID does not exist"
Nov 03 01:08:44.885711 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.885682    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") pod \"cd89036a-534f-479a-b030-6ee35f289901\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") "
Nov 03 01:08:44.885905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.885755    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\") pod \"cd89036a-534f-479a-b030-6ee35f289901\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") "
Nov 03 01:08:44.885905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.885802    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\" (UniqueName: \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\") pod \"cd89036a-534f-479a-b030-6ee35f289901\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") "
Nov 03 01:08:44.886055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.886035    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\") pod \"cd89036a-534f-479a-b030-6ee35f289901\" (UID: \"cd89036a-534f-479a-b030-6ee35f289901\") "
Nov 03 01:08:44.886462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.886427    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca" pod "cd89036a-534f-479a-b030-6ee35f289901" (UID: "cd89036a-534f-479a-b030-6ee35f289901"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:08:44.886879 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.886857    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd89036a-534f-479a-b030-6ee35f289901-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:08:44.887960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.887938    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs" pod "cd89036a-534f-479a-b030-6ee35f289901" (UID: "cd89036a-534f-479a-b030-6ee35f289901"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:08:44.888064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.887961    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67" pod "cd89036a-534f-479a-b030-6ee35f289901" (UID: "cd89036a-534f-479a-b030-6ee35f289901"). InnerVolumeSpecName "kube-api-access-sts67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:08:44.888205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.888186    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls" pod "cd89036a-534f-479a-b030-6ee35f289901" (UID: "cd89036a-534f-479a-b030-6ee35f289901"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:08:44.988197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.988146    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:08:44.988197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.988189    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd89036a-534f-479a-b030-6ee35f289901-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:08:44.988401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:44.988205    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sts67\" (UniqueName: \"kubernetes.io/projected/cd89036a-534f-479a-b030-6ee35f289901-kube-api-access-sts67\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:08:45.127792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:45.127758    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:08:45.132530 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:45.132500    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-84e377-f8768b479-st7vl"]
Nov 03 01:08:46.923096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:46.923061    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd89036a-534f-479a-b030-6ee35f289901" path="/var/lib/kubelet/pods/cd89036a-534f-479a-b030-6ee35f289901/volumes"
Nov 03 01:08:55.070580 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.070545    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7031-b93ad0"]
Nov 03 01:08:55.071248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.071232    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cd89036a-534f-479a-b030-6ee35f289901" containerName="maas-api"
Nov 03 01:08:55.071320 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.071248    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd89036a-534f-479a-b030-6ee35f289901" containerName="maas-api"
Nov 03 01:08:55.071382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.071374    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd89036a-534f-479a-b030-6ee35f289901" containerName="maas-api"
Nov 03 01:08:55.075378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.075356    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:08:55.083383 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.083352    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\""
Nov 03 01:08:55.083573 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.083511    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7031-b93ad0"]
Nov 03 01:08:55.204386 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.204346    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnbs\" (UniqueName: \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\") pod \"test-curl-7031-b93ad0\" (UID: \"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc\") " pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:08:55.311656 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.311609    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnbs\" (UniqueName: \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\") pod \"test-curl-7031-b93ad0\" (UID: \"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc\") " pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:08:55.320902 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.320837    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnbs\" (UniqueName: \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\") pod \"test-curl-7031-b93ad0\" (UID: \"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc\") " pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:08:55.398859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.398799    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:08:55.718376 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.718344    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7031-b93ad0"]
Nov 03 01:08:55.901321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:55.901288    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7031-b93ad0" event={"ID":"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc","Type":"ContainerStarted","Data":"759bfdead117c5340d84bd037a66f1e32effd0ad1d8b7f0f3bd83887c8bd9b59"}
Nov 03 01:08:57.919534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:57.919495    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7031-b93ad0" event={"ID":"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc","Type":"ContainerStarted","Data":"c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8"}
Nov 03 01:08:57.938584 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:57.938529    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7031-b93ad0" podStartSLOduration=1.740970463 podStartE2EDuration="2.938516677s" totalImagesPullingTime="1.197546214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:57.93320767 +0000 UTC m=+1683.146951335" watchObservedRunningTime="2026-11-03 01:08:57.938516677 +0000 UTC m=+1683.152260349"
Nov 03 01:08:58.629711 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:58.629674    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7031-b93ad0"]
Nov 03 01:08:58.635173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:58.635126    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7031-b93ad0"]
Nov 03 01:08:59.966126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:08:59.965725    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7031-b93ad0" podUID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" containerName="test-curl-7031-b93ad0" containerID="cri-o://c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8" gracePeriod=2
Nov 03 01:09:02.311772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.311744    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:09:02.404811 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.404777    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\" (UniqueName: \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\") pod \"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc\" (UID: \"0cd1d0f2-fbca-4a73-99e9-918038f7a5bc\") "
Nov 03 01:09:02.406801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.406771    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs" pod "0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" (UID: "0cd1d0f2-fbca-4a73-99e9-918038f7a5bc"). InnerVolumeSpecName "kube-api-access-btnbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:02.507404 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.507365    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnbs\" (UniqueName: \"kubernetes.io/projected/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc-kube-api-access-btnbs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:09:02.904878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.904842    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" path="/var/lib/kubelet/pods/0cd1d0f2-fbca-4a73-99e9-918038f7a5bc/volumes"
Nov 03 01:09:02.994042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.994005    4531 generic.go:356] "Generic (PLEG): container finished" podID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" containerID="c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8" exitCode=137
Nov 03 01:09:02.994256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.994050    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7031-b93ad0"
Nov 03 01:09:02.994256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:02.994082    4531 scope.go:122] "RemoveContainer" containerID="c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8"
Nov 03 01:09:03.004398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:03.004170    4531 scope.go:122] "RemoveContainer" containerID="c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8"
Nov 03 01:09:03.004492 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:03.004471    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8\": container with ID starting with c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8 not found: ID does not exist" containerID="c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8"
Nov 03 01:09:03.004536 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:03.004501    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8"} err="failed to get container status \"c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8\": rpc error: code = NotFound desc = could not find container \"c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8\": container with ID starting with c1e38c7fa32b7b4569860bdd7da4a8bdec3600d88ccaecab19b051a8196b62a8 not found: ID does not exist"
Nov 03 01:09:09.879393 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:09.879359    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:09:12.227186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.218194    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7028-d89cda"]
Nov 03 01:09:12.227186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.219570    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" containerName="test-curl-7031-b93ad0"
Nov 03 01:09:12.227186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.219590    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" containerName="test-curl-7031-b93ad0"
Nov 03 01:09:12.227186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.219740    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd1d0f2-fbca-4a73-99e9-918038f7a5bc" containerName="test-curl-7031-b93ad0"
Nov 03 01:09:12.248581 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.248546    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:12.253660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.253629    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\""
Nov 03 01:09:12.259399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.257511    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-d89cda"]
Nov 03 01:09:12.411195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.411140    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lml77\" (UniqueName: \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\") pod \"test-curl-7028-d89cda\" (UID: \"91a34055-ee12-4e92-ab41-2b32c075a269\") " pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:12.515402 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.515281    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lml77\" (UniqueName: \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\") pod \"test-curl-7028-d89cda\" (UID: \"91a34055-ee12-4e92-ab41-2b32c075a269\") " pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:12.531229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.526953    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lml77\" (UniqueName: \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\") pod \"test-curl-7028-d89cda\" (UID: \"91a34055-ee12-4e92-ab41-2b32c075a269\") " pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:12.582542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.582510    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:12.869309 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:12.869276    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-d89cda"]
Nov 03 01:09:12.869696 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:09:12.869670    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91a34055_ee12_4e92_ab41_2b32c075a269.slice/crio-83c09fa8016fe1f50c99cc0ebf3fd5382b270fdc1d68178d82157f6bc0ff85e6 WatchSource:0}: Error finding container 83c09fa8016fe1f50c99cc0ebf3fd5382b270fdc1d68178d82157f6bc0ff85e6: Status 404 returned error can't find the container with id 83c09fa8016fe1f50c99cc0ebf3fd5382b270fdc1d68178d82157f6bc0ff85e6
Nov 03 01:09:13.251666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:13.251579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-d89cda" event={"ID":"91a34055-ee12-4e92-ab41-2b32c075a269","Type":"ContainerStarted","Data":"6735d63927613b64df56e298e38e80085e697f9302222103806dbb2837dffd86"}
Nov 03 01:09:13.251666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:13.251631    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-d89cda" event={"ID":"91a34055-ee12-4e92-ab41-2b32c075a269","Type":"ContainerStarted","Data":"83c09fa8016fe1f50c99cc0ebf3fd5382b270fdc1d68178d82157f6bc0ff85e6"}
Nov 03 01:09:13.268960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:13.268912    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7028-d89cda" podStartSLOduration=1.26889886 podStartE2EDuration="1.26889886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:13.267001643 +0000 UTC m=+1698.480745318" watchObservedRunningTime="2026-11-03 01:09:13.26889886 +0000 UTC m=+1698.482642532"
Nov 03 01:09:13.909976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:13.909928    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7028-d89cda"]
Nov 03 01:09:13.915655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:13.915615    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7028-d89cda"]
Nov 03 01:09:14.054704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.054539    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7028-f5c007"]
Nov 03 01:09:14.055682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.055656    4531 cpu_manager.go:414] "containerMap: removing container" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" containerName="test-curl-7028-d89cda"
Nov 03 01:09:14.055682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.055681    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" containerName="test-curl-7028-d89cda"
Nov 03 01:09:14.059370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.055897    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" containerName="test-curl-7028-d89cda"
Nov 03 01:09:14.060431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.060407    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:14.060995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.060971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-f5c007"]
Nov 03 01:09:14.157861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.157819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j562\" (UniqueName: \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\") pod \"test-curl-7028-f5c007\" (UID: \"9f19ce08-ede2-4114-a909-79333465ef25\") " pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:14.259190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.259084    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j562\" (UniqueName: \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\") pod \"test-curl-7028-f5c007\" (UID: \"9f19ce08-ede2-4114-a909-79333465ef25\") " pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:14.268195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.268129    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j562\" (UniqueName: \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\") pod \"test-curl-7028-f5c007\" (UID: \"9f19ce08-ede2-4114-a909-79333465ef25\") " pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:14.380997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.380958    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:14.672205 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:09:14.666860    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f19ce08_ede2_4114_a909_79333465ef25.slice/crio-5db4be2788475538f3fb73c3e320480a67457946cee5ca19806b24b2cdeb94a9 WatchSource:0}: Error finding container 5db4be2788475538f3fb73c3e320480a67457946cee5ca19806b24b2cdeb94a9: Status 404 returned error can't find the container with id 5db4be2788475538f3fb73c3e320480a67457946cee5ca19806b24b2cdeb94a9
Nov 03 01:09:14.672205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:14.667977    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-f5c007"]
Nov 03 01:09:15.261890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.261766    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:09:15.267917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.267884    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.279022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.275370    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-531447-serving-cert\""
Nov 03 01:09:15.279459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.279424    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7028-d89cda" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" containerName="test-curl-7028-d89cda" containerID="cri-o://6735d63927613b64df56e298e38e80085e697f9302222103806dbb2837dffd86" gracePeriod=2
Nov 03 01:09:15.280885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.280558    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-f5c007" event={"ID":"9f19ce08-ede2-4114-a909-79333465ef25","Type":"ContainerStarted","Data":"41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9"}
Nov 03 01:09:15.280885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.280607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:09:15.281018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.280992    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-f5c007" event={"ID":"9f19ce08-ede2-4114-a909-79333465ef25","Type":"ContainerStarted","Data":"5db4be2788475538f3fb73c3e320480a67457946cee5ca19806b24b2cdeb94a9"}
Nov 03 01:09:15.303072 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.303039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwrb\" (UniqueName: \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.303248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.303092    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.303347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.303326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.303430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.303414    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.310140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.310100    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7028-f5c007" podStartSLOduration=1.3100885199999999 podStartE2EDuration="1.31008852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:15.309409968 +0000 UTC m=+1700.523153639" watchObservedRunningTime="2026-11-03 01:09:15.31008852 +0000 UTC m=+1700.523832225"
Nov 03 01:09:15.405593 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.405547    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwrb\" (UniqueName: \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.405911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.405606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.405911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.405763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.410178 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.406483    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.410178 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:15.407223    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-serving-cert: secret "maas-api-e2e-shared-a-w2-531447-serving-cert" not found
Nov 03 01:09:15.410178 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:15.407395    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls podName:e018a29c-f897-4e40-a2fa-94cb5b4c56a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:15.90727421 +0000 UTC m=+1701.121017865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls") pod "maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" (UID: "e018a29c-f897-4e40-a2fa-94cb5b4c56a9") : secret "maas-api-e2e-shared-a-w2-531447-serving-cert" not found
Nov 03 01:09:15.410178 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.408526    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.432248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.428147    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.432248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.431939    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwrb\" (UniqueName: \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.914233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.914148    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.917032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.916999    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:15.965858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.958412    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7028-f5c007"]
Nov 03 01:09:15.965858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:15.965594    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7028-f5c007"]
Nov 03 01:09:16.191305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.187337    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7028-0c2035"]
Nov 03 01:09:16.191305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.188518    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9f19ce08-ede2-4114-a909-79333465ef25" containerName="test-curl-7028-f5c007"
Nov 03 01:09:16.191305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.188545    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f19ce08-ede2-4114-a909-79333465ef25" containerName="test-curl-7028-f5c007"
Nov 03 01:09:16.191305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.188860    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f19ce08-ede2-4114-a909-79333465ef25" containerName="test-curl-7028-f5c007"
Nov 03 01:09:16.208196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.196475    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-0c2035"]
Nov 03 01:09:16.208196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.196605    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:16.212772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.210588    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:16.298194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.296912    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbcd\" (UniqueName: \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\") pod \"test-curl-7028-0c2035\" (UID: \"1ac3d741-0220-4e53-8eb5-6c3f598062a2\") " pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:16.400211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.399552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbcd\" (UniqueName: \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\") pod \"test-curl-7028-0c2035\" (UID: \"1ac3d741-0220-4e53-8eb5-6c3f598062a2\") " pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:16.409371 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.409344    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbcd\" (UniqueName: \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\") pod \"test-curl-7028-0c2035\" (UID: \"1ac3d741-0220-4e53-8eb5-6c3f598062a2\") " pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:16.622727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.606828    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:16.711749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.711467    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:09:16.947208 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:09:16.947172    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ac3d741_0220_4e53_8eb5_6c3f598062a2.slice/crio-1a1f4243717d6506f039c2ea715ba02cb8f2004aeb6b3e2f92bb35a0f82c2f63 WatchSource:0}: Error finding container 1a1f4243717d6506f039c2ea715ba02cb8f2004aeb6b3e2f92bb35a0f82c2f63: Status 404 returned error can't find the container with id 1a1f4243717d6506f039c2ea715ba02cb8f2004aeb6b3e2f92bb35a0f82c2f63
Nov 03 01:09:16.949777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:16.949751    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-0c2035"]
Nov 03 01:09:17.346615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.346578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" event={"ID":"e018a29c-f897-4e40-a2fa-94cb5b4c56a9","Type":"ContainerStarted","Data":"608835b83e85d6367411bb72387e3b2a9bc3cfdc4fabc8ceccc5cab50eef6f10"}
Nov 03 01:09:17.346997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.346648    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" event={"ID":"e018a29c-f897-4e40-a2fa-94cb5b4c56a9","Type":"ContainerStarted","Data":"da20c4222152539bd0e1dee21dbff5d081df4da9258aabcb1ed14010778f78d3"}
Nov 03 01:09:17.346997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.346679    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:17.349253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.349210    4531 generic.go:356] "Generic (PLEG): container finished" podID="91a34055-ee12-4e92-ab41-2b32c075a269" containerID="6735d63927613b64df56e298e38e80085e697f9302222103806dbb2837dffd86" exitCode=137
Nov 03 01:09:17.352632 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.352588    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-0c2035" event={"ID":"1ac3d741-0220-4e53-8eb5-6c3f598062a2","Type":"ContainerStarted","Data":"97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543"}
Nov 03 01:09:17.352632 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.352634    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-0c2035" event={"ID":"1ac3d741-0220-4e53-8eb5-6c3f598062a2","Type":"ContainerStarted","Data":"1a1f4243717d6506f039c2ea715ba02cb8f2004aeb6b3e2f92bb35a0f82c2f63"}
Nov 03 01:09:17.352962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.352931    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7028-f5c007" podUID="9f19ce08-ede2-4114-a909-79333465ef25" containerName="test-curl-7028-f5c007" containerID="cri-o://41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9" gracePeriod=2
Nov 03 01:09:17.457271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.454143    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" podStartSLOduration=1.9381783320000001 podStartE2EDuration="2.454124546s" totalImagesPullingTime="515.946214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:17.38028013 +0000 UTC m=+1702.594023803" watchObservedRunningTime="2026-11-03 01:09:17.454124546 +0000 UTC m=+1702.667868215"
Nov 03 01:09:17.459210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.459140    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7028-0c2035" podStartSLOduration=1.459122977 podStartE2EDuration="1.459122977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:17.431402763 +0000 UTC m=+1702.645146436" watchObservedRunningTime="2026-11-03 01:09:17.459122977 +0000 UTC m=+1702.672866651"
Nov 03 01:09:17.674635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.674538    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:17.710680 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.710647    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\" (UniqueName: \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\") pod \"91a34055-ee12-4e92-ab41-2b32c075a269\" (UID: \"91a34055-ee12-4e92-ab41-2b32c075a269\") "
Nov 03 01:09:17.712804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.712769    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77" pod "91a34055-ee12-4e92-ab41-2b32c075a269" (UID: "91a34055-ee12-4e92-ab41-2b32c075a269"). InnerVolumeSpecName "kube-api-access-lml77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:17.813524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:17.813439    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lml77\" (UniqueName: \"kubernetes.io/projected/91a34055-ee12-4e92-ab41-2b32c075a269-kube-api-access-lml77\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:09:18.094561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.094523    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7028-0c2035"]
Nov 03 01:09:18.099071 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.099040    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7028-0c2035"]
Nov 03 01:09:18.238468 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.237954    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7028-86b699"]
Nov 03 01:09:18.239195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.239154    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" containerName="test-curl-7028-0c2035"
Nov 03 01:09:18.239315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.239196    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" containerName="test-curl-7028-0c2035"
Nov 03 01:09:18.239414 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.239393    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" containerName="test-curl-7028-0c2035"
Nov 03 01:09:18.243118 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.243093    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:18.245144 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.245122    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-86b699"]
Nov 03 01:09:18.331853 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.331791    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c549f\" (UniqueName: \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\") pod \"test-curl-7028-86b699\" (UID: \"51c242d5-8cb7-4ffb-b8be-f2f7734d352a\") " pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:18.373413 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.367638    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:18.373413 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.370740    4531 scope.go:122] "RemoveContainer" containerID="6735d63927613b64df56e298e38e80085e697f9302222103806dbb2837dffd86"
Nov 03 01:09:18.382596 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:18.382562    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-d89cda\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" pod="opendatahub/test-curl-7028-d89cda"
Nov 03 01:09:18.433872 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.433834    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c549f\" (UniqueName: \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\") pod \"test-curl-7028-86b699\" (UID: \"51c242d5-8cb7-4ffb-b8be-f2f7734d352a\") " pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:18.441465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.441436    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c549f\" (UniqueName: \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\") pod \"test-curl-7028-86b699\" (UID: \"51c242d5-8cb7-4ffb-b8be-f2f7734d352a\") " pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:18.566505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.566472    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:18.853541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.853512    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7028-86b699"]
Nov 03 01:09:18.856588 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:09:18.856551    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c242d5_8cb7_4ffb_b8be_f2f7734d352a.slice/crio-66552417517f38a038d72cd82b23725514dfc9f6cd3d8245453fe4897c3c2df5 WatchSource:0}: Error finding container 66552417517f38a038d72cd82b23725514dfc9f6cd3d8245453fe4897c3c2df5: Status 404 returned error can't find the container with id 66552417517f38a038d72cd82b23725514dfc9f6cd3d8245453fe4897c3c2df5
Nov 03 01:09:18.906311 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:18.906282    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a34055-ee12-4e92-ab41-2b32c075a269" path="/var/lib/kubelet/pods/91a34055-ee12-4e92-ab41-2b32c075a269/volumes"
Nov 03 01:09:19.396889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.396843    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7028-0c2035" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" containerName="test-curl-7028-0c2035" containerID="cri-o://97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543" gracePeriod=2
Nov 03 01:09:19.404099 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.398028    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-86b699" event={"ID":"51c242d5-8cb7-4ffb-b8be-f2f7734d352a","Type":"ContainerStarted","Data":"ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313"}
Nov 03 01:09:19.404099 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.398076    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7028-86b699" event={"ID":"51c242d5-8cb7-4ffb-b8be-f2f7734d352a","Type":"ContainerStarted","Data":"66552417517f38a038d72cd82b23725514dfc9f6cd3d8245453fe4897c3c2df5"}
Nov 03 01:09:19.421474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.419516    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7028-86b699" podStartSLOduration=1.4194941540000001 podStartE2EDuration="1.419494154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:19.416520922 +0000 UTC m=+1704.630264594" watchObservedRunningTime="2026-11-03 01:09:19.419494154 +0000 UTC m=+1704.633237841"
Nov 03 01:09:19.634210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.634082    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:19.768113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.768081    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\" (UniqueName: \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\") pod \"9f19ce08-ede2-4114-a909-79333465ef25\" (UID: \"9f19ce08-ede2-4114-a909-79333465ef25\") "
Nov 03 01:09:19.770795 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.770762    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562" pod "9f19ce08-ede2-4114-a909-79333465ef25" (UID: "9f19ce08-ede2-4114-a909-79333465ef25"). InnerVolumeSpecName "kube-api-access-5j562". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:19.870808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:19.870771    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j562\" (UniqueName: \"kubernetes.io/projected/9f19ce08-ede2-4114-a909-79333465ef25-kube-api-access-5j562\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:09:20.025314 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.025211    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7028-86b699"]
Nov 03 01:09:20.031040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.030997    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7028-86b699"]
Nov 03 01:09:20.405479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.405445    4531 generic.go:356] "Generic (PLEG): container finished" podID="9f19ce08-ede2-4114-a909-79333465ef25" containerID="41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9" exitCode=137
Nov 03 01:09:20.405980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.405503    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:20.405980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.405559    4531 scope.go:122] "RemoveContainer" containerID="41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9"
Nov 03 01:09:20.418051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.418033    4531 scope.go:122] "RemoveContainer" containerID="41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9"
Nov 03 01:09:20.418423 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:20.418395    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9\": container with ID starting with 41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9 not found: ID does not exist" containerID="41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9"
Nov 03 01:09:20.418485 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.418439    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9"} err="failed to get container status \"41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9\": rpc error: code = NotFound desc = could not find container \"41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9\": container with ID starting with 41a5c2b23fbbea00eb0e4c9a2c65219ff2f390e71eede4162f97cfbcdf8fb8e9 not found: ID does not exist"
Nov 03 01:09:20.419978 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:20.419956    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-f5c007\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="9f19ce08-ede2-4114-a909-79333465ef25" pod="opendatahub/test-curl-7028-f5c007"
Nov 03 01:09:20.904710 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:20.904679    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f19ce08-ede2-4114-a909-79333465ef25" path="/var/lib/kubelet/pods/9f19ce08-ede2-4114-a909-79333465ef25/volumes"
Nov 03 01:09:21.414701 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:21.414669    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7028-86b699" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" containerName="test-curl-7028-86b699" containerID="cri-o://ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313" gracePeriod=2
Nov 03 01:09:21.661635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:21.661604    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:21.805260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:21.805141    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\" (UniqueName: \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\") pod \"1ac3d741-0220-4e53-8eb5-6c3f598062a2\" (UID: \"1ac3d741-0220-4e53-8eb5-6c3f598062a2\") "
Nov 03 01:09:21.807290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:21.807255    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd" pod "1ac3d741-0220-4e53-8eb5-6c3f598062a2" (UID: "1ac3d741-0220-4e53-8eb5-6c3f598062a2"). InnerVolumeSpecName "kube-api-access-wlbcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:21.906961 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:21.906929    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbcd\" (UniqueName: \"kubernetes.io/projected/1ac3d741-0220-4e53-8eb5-6c3f598062a2-kube-api-access-wlbcd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:09:22.462254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.445756    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" containerID="97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543" exitCode=137
Nov 03 01:09:22.462254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.445972    4531 scope.go:122] "RemoveContainer" containerID="97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543"
Nov 03 01:09:22.462254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.446134    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:22.476400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.474778    4531 scope.go:122] "RemoveContainer" containerID="97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543"
Nov 03 01:09:22.476400 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:22.475204    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543\": container with ID starting with 97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543 not found: ID does not exist" containerID="97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543"
Nov 03 01:09:22.476400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.475243    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543"} err="failed to get container status \"97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543\": rpc error: code = NotFound desc = could not find container \"97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543\": container with ID starting with 97fd69defe21994c8234e90c74434c6f9d4c6a63527f55f9c5f835e2b869b543 not found: ID does not exist"
Nov 03 01:09:22.480698 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:22.480667    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-0c2035\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" pod="opendatahub/test-curl-7028-0c2035"
Nov 03 01:09:22.909618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:22.909576    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac3d741-0220-4e53-8eb5-6c3f598062a2" path="/var/lib/kubelet/pods/1ac3d741-0220-4e53-8eb5-6c3f598062a2/volumes"
Nov 03 01:09:23.374736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:23.374706    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:09:23.774272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:23.774241    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:23.868945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:23.868897    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\" (UniqueName: \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\") pod \"51c242d5-8cb7-4ffb-b8be-f2f7734d352a\" (UID: \"51c242d5-8cb7-4ffb-b8be-f2f7734d352a\") "
Nov 03 01:09:23.871269 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:23.871226    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f" pod "51c242d5-8cb7-4ffb-b8be-f2f7734d352a" (UID: "51c242d5-8cb7-4ffb-b8be-f2f7734d352a"). InnerVolumeSpecName "kube-api-access-c549f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:23.970742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:23.970623    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c549f\" (UniqueName: \"kubernetes.io/projected/51c242d5-8cb7-4ffb-b8be-f2f7734d352a-kube-api-access-c549f\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:09:24.470321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.470287    4531 generic.go:356] "Generic (PLEG): container finished" podID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" containerID="ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313" exitCode=137
Nov 03 01:09:24.470523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.470342    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:24.470523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.470348    4531 scope.go:122] "RemoveContainer" containerID="ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313"
Nov 03 01:09:24.482526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.482506    4531 scope.go:122] "RemoveContainer" containerID="ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313"
Nov 03 01:09:24.482611 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:24.482560    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-86b699\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:24.482832 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:24.482812    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313\": container with ID starting with ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313 not found: ID does not exist" containerID="ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313"
Nov 03 01:09:24.482888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.482841    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313"} err="failed to get container status \"ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313\": rpc error: code = NotFound desc = could not find container \"ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313\": container with ID starting with ea9df77efbbe3e35e99faf205af9be6794acb9020d95bb9d0f210fde265e1313 not found: ID does not exist"
Nov 03 01:09:24.929199 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:24.926454    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7028-86b699\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" pod="opendatahub/test-curl-7028-86b699"
Nov 03 01:09:24.945637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:24.945607    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" path="/var/lib/kubelet/pods/51c242d5-8cb7-4ffb-b8be-f2f7734d352a/volumes"
Nov 03 01:09:38.619074 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.619038    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:09:38.620886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.620858    4531 cpu_manager.go:414] "containerMap: removing container" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" containerName="test-curl-7028-86b699"
Nov 03 01:09:38.620886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.620887    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" containerName="test-curl-7028-86b699"
Nov 03 01:09:38.621435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.621355    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c242d5-8cb7-4ffb-b8be-f2f7734d352a" containerName="test-curl-7028-86b699"
Nov 03 01:09:38.641113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.641081    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.643093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.643064    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:09:38.658526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.658497    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-821f5a-serving-cert\""
Nov 03 01:09:38.832417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.828461    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.832417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.828551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9lx\" (UniqueName: \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.832417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.828752    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.832417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.828965    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.931017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.930918    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.931315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.931291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9lx\" (UniqueName: \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.931652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.931630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.931932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.931899    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.932976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.932942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.941342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.935588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.941342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.937929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.945207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.945177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9lx\" (UniqueName: \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\") pod \"maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:38.969778 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:38.969744    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:39.283072 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:09:39.283037    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1aeaf6_086e_49dc_900c_18870ba9ae0c.slice/crio-079096b84eb9ae9054af1f389427cd766cadfaece6df1e735e36962246e7c3e2 WatchSource:0}: Error finding container 079096b84eb9ae9054af1f389427cd766cadfaece6df1e735e36962246e7c3e2: Status 404 returned error can't find the container with id 079096b84eb9ae9054af1f389427cd766cadfaece6df1e735e36962246e7c3e2
Nov 03 01:09:39.283503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:39.283478    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:09:39.660210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:39.655770    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" event={"ID":"9a1aeaf6-086e-49dc-900c-18870ba9ae0c","Type":"ContainerStarted","Data":"079096b84eb9ae9054af1f389427cd766cadfaece6df1e735e36962246e7c3e2"}
Nov 03 01:09:39.934742 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:09:39.934650    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:09:40.675237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:40.675189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" event={"ID":"9a1aeaf6-086e-49dc-900c-18870ba9ae0c","Type":"ContainerStarted","Data":"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"}
Nov 03 01:09:40.676489 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:40.676459    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:40.694207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:40.694127    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" podStartSLOduration=2.20592302 podStartE2EDuration="2.69410717s" totalImagesPullingTime="488.18415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:40.692455159 +0000 UTC m=+1725.906198913" watchObservedRunningTime="2026-11-03 01:09:40.69410717 +0000 UTC m=+1725.907850845"
Nov 03 01:09:47.152908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.152867    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:09:47.176303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.160936    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.176303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.170003    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ef434a-serving-cert\""
Nov 03 01:09:47.176303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.170215    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:09:47.278446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.278408    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxswx\" (UniqueName: \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.278682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.278598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.278797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.278779    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.278921 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.278903    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.379875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.379836    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.380053 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.379904    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxswx\" (UniqueName: \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.380173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.380111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.380317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.380301    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.380978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.380955    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.382559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.382534    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.383575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.383552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.388976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.388949    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxswx\" (UniqueName: \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\") pod \"maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.496783 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.496697    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:47.712605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.712572    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:09:47.814915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:47.814877    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:09:48.799575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:48.799544    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" event={"ID":"21649128-15d0-4cf6-b0a1-08bc8991951f","Type":"ContainerStarted","Data":"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"}
Nov 03 01:09:48.799995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:48.799595    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" event={"ID":"21649128-15d0-4cf6-b0a1-08bc8991951f","Type":"ContainerStarted","Data":"6433e43a0dd8dc8b044a3db0fb896fff09d05218c33426158bc870062f70e575"}
Nov 03 01:09:48.799995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:48.799686    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:09:48.818609 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:48.818544    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" podStartSLOduration=1.345112473 podStartE2EDuration="1.818525346s" totalImagesPullingTime="473.412873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:48.817519037 +0000 UTC m=+1734.031262709" watchObservedRunningTime="2026-11-03 01:09:48.818525346 +0000 UTC m=+1734.032269018"
Nov 03 01:09:54.815119 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:09:54.815085    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:10:09.882819 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:09.882726    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:10:11.589189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.585609    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7037-279bc9"]
Nov 03 01:10:11.658412 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.651488    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:11.658412 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.651380    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-279bc9"]
Nov 03 01:10:11.662912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.662310    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\""
Nov 03 01:10:11.831724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.831682    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6zh\" (UniqueName: \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\") pod \"test-curl-7037-279bc9\" (UID: \"ac3a2172-1b6b-4832-b371-30b0e8f91920\") " pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:11.935204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.933973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6zh\" (UniqueName: \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\") pod \"test-curl-7037-279bc9\" (UID: \"ac3a2172-1b6b-4832-b371-30b0e8f91920\") " pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:11.946404 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.945986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6zh\" (UniqueName: \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\") pod \"test-curl-7037-279bc9\" (UID: \"ac3a2172-1b6b-4832-b371-30b0e8f91920\") " pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:11.994684 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:11.994412    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:12.291699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:12.291642    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7037-279bc9"]
Nov 03 01:10:13.351183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:13.350709    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-279bc9" event={"ID":"ac3a2172-1b6b-4832-b371-30b0e8f91920","Type":"ContainerStarted","Data":"cc40b8736e1b9a10439244ae0fd307d10f89c8050087388324b52a89fa6e7fa9"}
Nov 03 01:10:13.351183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:13.350764    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7037-279bc9" event={"ID":"ac3a2172-1b6b-4832-b371-30b0e8f91920","Type":"ContainerStarted","Data":"c4067419f4a5a47a73c72d5c05489bc117ec795900888ec7f73b58793affc664"}
Nov 03 01:10:13.390186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:13.389856    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7037-279bc9" podStartSLOduration=2.389831526 podStartE2EDuration="2.389831526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:13.37736782 +0000 UTC m=+1758.591111493" watchObservedRunningTime="2026-11-03 01:10:13.389831526 +0000 UTC m=+1758.603575201"
Nov 03 01:10:14.045452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:14.045415    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7037-279bc9"]
Nov 03 01:10:14.049909 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:14.049871    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7037-279bc9"]
Nov 03 01:10:15.368566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:15.368490    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7037-279bc9" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" containerName="test-curl-7037-279bc9" containerID="cri-o://cc40b8736e1b9a10439244ae0fd307d10f89c8050087388324b52a89fa6e7fa9" gracePeriod=2
Nov 03 01:10:17.469386 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:17.469043    4531 generic.go:356] "Generic (PLEG): container finished" podID="ac3a2172-1b6b-4832-b371-30b0e8f91920" containerID="cc40b8736e1b9a10439244ae0fd307d10f89c8050087388324b52a89fa6e7fa9" exitCode=137
Nov 03 01:10:17.591409 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:17.591384    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:17.700258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:17.700148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\" (UniqueName: \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\") pod \"ac3a2172-1b6b-4832-b371-30b0e8f91920\" (UID: \"ac3a2172-1b6b-4832-b371-30b0e8f91920\") "
Nov 03 01:10:17.702890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:17.702855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh" pod "ac3a2172-1b6b-4832-b371-30b0e8f91920" (UID: "ac3a2172-1b6b-4832-b371-30b0e8f91920"). InnerVolumeSpecName "kube-api-access-mc6zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:17.802009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:17.801976    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6zh\" (UniqueName: \"kubernetes.io/projected/ac3a2172-1b6b-4832-b371-30b0e8f91920-kube-api-access-mc6zh\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:10:18.496714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:18.496683    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:18.497210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:18.496734    4531 scope.go:122] "RemoveContainer" containerID="cc40b8736e1b9a10439244ae0fd307d10f89c8050087388324b52a89fa6e7fa9"
Nov 03 01:10:18.508833 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:18.508802    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7037-279bc9\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" pod="opendatahub/test-curl-7037-279bc9"
Nov 03 01:10:18.904991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:18.904958    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" path="/var/lib/kubelet/pods/ac3a2172-1b6b-4832-b371-30b0e8f91920/volumes"
Nov 03 01:10:22.408653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.408615    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:10:22.409693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.409676    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" containerName="test-curl-7037-279bc9"
Nov 03 01:10:22.409765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.409695    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" containerName="test-curl-7037-279bc9"
Nov 03 01:10:22.409852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.409840    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3a2172-1b6b-4832-b371-30b0e8f91920" containerName="test-curl-7037-279bc9"
Nov 03 01:10:22.416386 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.416359    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.418541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.418514    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:10:22.428726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.428700    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-af8244-serving-cert\""
Nov 03 01:10:22.500131 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.500098    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.500585 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.500557    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.500682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.500609    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.500682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.500653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z28lv\" (UniqueName: \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.606475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.606429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.610716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.610686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.614378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.614343    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.614517 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.614419    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.614517 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.614481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z28lv\" (UniqueName: \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.616944 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.616914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:22.617099 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:22.617080    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-serving-cert: secret "maas-api-e2e-extmod-b-af8244-serving-cert" not found
Nov 03 01:10:22.617213 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:22.617198    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls podName:5922c0ec-e00e-4d6a-b0a6-08ff53bbb778 nodeName:}" failed. No retries permitted until 2026-11-03 01:10:23.1171491 +0000 UTC m=+1768.330892765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls") pod "maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" (UID: "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778") : secret "maas-api-e2e-extmod-b-af8244-serving-cert" not found
Nov 03 01:10:22.625102 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:22.625074    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z28lv\" (UniqueName: \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:23.123236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:23.123201    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:23.125761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:23.125733    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:23.353184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:23.352805    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:23.756149 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:23.756117    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:10:23.756833 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:10:23.756811    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5922c0ec_e00e_4d6a_b0a6_08ff53bbb778.slice/crio-ff8e3c7bf6e6b920bd3230795c23ca4c403fb4d5c2e128812c0a6f3d09da72f8 WatchSource:0}: Error finding container ff8e3c7bf6e6b920bd3230795c23ca4c403fb4d5c2e128812c0a6f3d09da72f8: Status 404 returned error can't find the container with id ff8e3c7bf6e6b920bd3230795c23ca4c403fb4d5c2e128812c0a6f3d09da72f8
Nov 03 01:10:23.758317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:23.758297    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:10:24.764703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:24.764603    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" event={"ID":"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778","Type":"ContainerStarted","Data":"d49823d77d876a1431f8c73262e1c689c8f6161b4cbbf7e1f40b54a626562467"}
Nov 03 01:10:24.764703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:24.764661    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:24.764703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:24.764680    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" event={"ID":"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778","Type":"ContainerStarted","Data":"ff8e3c7bf6e6b920bd3230795c23ca4c403fb4d5c2e128812c0a6f3d09da72f8"}
Nov 03 01:10:24.789116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:24.789051    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" podStartSLOduration=2.213859119 podStartE2EDuration="2.789034025s" totalImagesPullingTime="575.174906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:24.785831093 +0000 UTC m=+1769.999574765" watchObservedRunningTime="2026-11-03 01:10:24.789034025 +0000 UTC m=+1770.002777703"
Nov 03 01:10:28.202687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.202644    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:10:28.207650 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.207618    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.220697 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.218729    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-027c19-serving-cert\""
Nov 03 01:10:28.243670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.243636    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:10:28.327259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.327223    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.327440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.327394    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5bk\" (UniqueName: \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.327527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.327504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.327573 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.327557    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.429034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.428991    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.433293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.429408    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5bk\" (UniqueName: \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.433653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.433621    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.433899 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.433876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.438826 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.438797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.465473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.461762    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5bk\" (UniqueName: \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.465473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.461904    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.474971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.474252    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\") pod \"maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.532821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.532793    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:28.833967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:28.833937    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:10:28.835524 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:10:28.835493    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555ec9eb_2fab_419c_86c1_0db07e23907d.slice/crio-2778a2ca0cfc746bd0dfa6d3d9f4a7b13fab03a6ded5b93d288f0b528f19c528 WatchSource:0}: Error finding container 2778a2ca0cfc746bd0dfa6d3d9f4a7b13fab03a6ded5b93d288f0b528f19c528: Status 404 returned error can't find the container with id 2778a2ca0cfc746bd0dfa6d3d9f4a7b13fab03a6ded5b93d288f0b528f19c528
Nov 03 01:10:29.813300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:29.813210    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" event={"ID":"555ec9eb-2fab-419c-86c1-0db07e23907d","Type":"ContainerStarted","Data":"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"}
Nov 03 01:10:29.813300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:29.813282    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" event={"ID":"555ec9eb-2fab-419c-86c1-0db07e23907d","Type":"ContainerStarted","Data":"2778a2ca0cfc746bd0dfa6d3d9f4a7b13fab03a6ded5b93d288f0b528f19c528"}
Nov 03 01:10:29.813716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:29.813312    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:29.833089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:29.833040    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" podStartSLOduration=1.341082648 podStartE2EDuration="1.83302432s" totalImagesPullingTime="491.941672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:29.82839665 +0000 UTC m=+1775.042140322" watchObservedRunningTime="2026-11-03 01:10:29.83302432 +0000 UTC m=+1775.046767991"
Nov 03 01:10:30.794258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:30.794216    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:10:35.844215 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:35.844180    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:10:37.427798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.427761    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:10:37.433820 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.433792    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.444564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.444536    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-7c759a-serving-cert\""
Nov 03 01:10:37.470254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.470223    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:10:37.603183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.601418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.603183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.601514    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbptp\" (UniqueName: \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.603183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.601564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.603183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.601739    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.703474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.703389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.703474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.703443    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbptp\" (UniqueName: \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.703723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.703479    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.703723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.703550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.705097 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.705070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.712274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.712140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbptp\" (UniqueName: \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.723521 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.723479    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.724765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.724729    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:37.779590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:37.779554    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:38.540646 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:38.537768    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:10:39.215737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:39.215579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" event={"ID":"341945e9-c2a6-4f1f-af71-647d40dc918e","Type":"ContainerStarted","Data":"8e40b11c359dd697c75e473191d62f85a17ad5bc641104fb9762bbfc9fbbcd54"}
Nov 03 01:10:39.215737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:39.215653    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" event={"ID":"341945e9-c2a6-4f1f-af71-647d40dc918e","Type":"ContainerStarted","Data":"706c7f58c66f04fb67637409def52e7299410f41c323203c0fdba3a5a52d5667"}
Nov 03 01:10:39.215737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:39.215699    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:39.260181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:39.244464    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" podStartSLOduration=1.754329073 podStartE2EDuration="2.244440826s" totalImagesPullingTime="490.111753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:39.234304741 +0000 UTC m=+1784.448048438" watchObservedRunningTime="2026-11-03 01:10:39.244440826 +0000 UTC m=+1784.458184498"
Nov 03 01:10:40.073497 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:40.073387    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:10:43.676215 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.676180    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7034-aaa247"]
Nov 03 01:10:43.680938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.680918    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:43.685145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.685121    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7l8r6\""
Nov 03 01:10:43.687785 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.687761    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-aaa247"]
Nov 03 01:10:43.785995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.785955    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggr6\" (UniqueName: \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\") pod \"test-curl-7034-aaa247\" (UID: \"14f47cad-ca64-4a68-8e8a-78ebaa856335\") " pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:43.887983 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.887947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggr6\" (UniqueName: \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\") pod \"test-curl-7034-aaa247\" (UID: \"14f47cad-ca64-4a68-8e8a-78ebaa856335\") " pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:43.896366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:43.896335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggr6\" (UniqueName: \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\") pod \"test-curl-7034-aaa247\" (UID: \"14f47cad-ca64-4a68-8e8a-78ebaa856335\") " pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:44.007514 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:44.007416    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:44.476482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:44.476451    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7034-aaa247"]
Nov 03 01:10:44.477622 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:10:44.477586    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14f47cad_ca64_4a68_8e8a_78ebaa856335.slice/crio-7795d2876205e6fff77c25b79cb1e68eb2f96c587f9f320f32007abfaf28f5c3 WatchSource:0}: Error finding container 7795d2876205e6fff77c25b79cb1e68eb2f96c587f9f320f32007abfaf28f5c3: Status 404 returned error can't find the container with id 7795d2876205e6fff77c25b79cb1e68eb2f96c587f9f320f32007abfaf28f5c3
Nov 03 01:10:45.243022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:45.242991    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:10:45.308596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:45.308542    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-aaa247" event={"ID":"14f47cad-ca64-4a68-8e8a-78ebaa856335","Type":"ContainerStarted","Data":"9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0"}
Nov 03 01:10:45.308791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:45.308621    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7034-aaa247" event={"ID":"14f47cad-ca64-4a68-8e8a-78ebaa856335","Type":"ContainerStarted","Data":"7795d2876205e6fff77c25b79cb1e68eb2f96c587f9f320f32007abfaf28f5c3"}
Nov 03 01:10:45.330924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:45.330860    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7034-aaa247" podStartSLOduration=2.330841727 podStartE2EDuration="2.330841727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:45.326465681 +0000 UTC m=+1790.540209352" watchObservedRunningTime="2026-11-03 01:10:45.330841727 +0000 UTC m=+1790.544585400"
Nov 03 01:10:46.063126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:46.063080    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7034-aaa247"]
Nov 03 01:10:46.077239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:46.073962    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7034-aaa247"]
Nov 03 01:10:47.328124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:47.328076    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7034-aaa247" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" containerName="test-curl-7034-aaa247" containerID="cri-o://9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0" gracePeriod=2
Nov 03 01:10:49.640838 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:49.640806    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:49.678440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:49.675060    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\" (UniqueName: \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\") pod \"14f47cad-ca64-4a68-8e8a-78ebaa856335\" (UID: \"14f47cad-ca64-4a68-8e8a-78ebaa856335\") "
Nov 03 01:10:49.707619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:49.706604    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6" pod "14f47cad-ca64-4a68-8e8a-78ebaa856335" (UID: "14f47cad-ca64-4a68-8e8a-78ebaa856335"). InnerVolumeSpecName "kube-api-access-jggr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:49.784734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:49.784701    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jggr6\" (UniqueName: \"kubernetes.io/projected/14f47cad-ca64-4a68-8e8a-78ebaa856335-kube-api-access-jggr6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:10:50.377210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.373818    4531 generic.go:356] "Generic (PLEG): container finished" podID="14f47cad-ca64-4a68-8e8a-78ebaa856335" containerID="9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0" exitCode=137
Nov 03 01:10:50.377210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.374135    4531 scope.go:122] "RemoveContainer" containerID="9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0"
Nov 03 01:10:50.382852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.382811    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:50.400382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.400112    4531 scope.go:122] "RemoveContainer" containerID="9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0"
Nov 03 01:10:50.400625 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:50.400604    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0\": container with ID starting with 9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0 not found: ID does not exist" containerID="9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0"
Nov 03 01:10:50.400715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.400638    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0"} err="failed to get container status \"9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0\": rpc error: code = NotFound desc = could not find container \"9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0\": container with ID starting with 9d37a7197c7810e77b7dec06ea991727f444c09a52683f5868cd2858662862f0 not found: ID does not exist"
Nov 03 01:10:50.411176 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:10:50.411135    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7034-aaa247\" is forbidden: User \"system:node:ip-10-0-136-112.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-112.us-east-2.compute.internal' and this object" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" pod="opendatahub/test-curl-7034-aaa247"
Nov 03 01:10:50.970559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:50.968029    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" path="/var/lib/kubelet/pods/14f47cad-ca64-4a68-8e8a-78ebaa856335/volumes"
Nov 03 01:10:55.258059 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:55.258022    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 01:10:55.258541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:10:55.258272    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy" containerID="cri-o://a46911c5d43a04bd1d33c9b1a146cfe2ea65482868e63527c6d247964e5176ab" gracePeriod=30
Nov 03 01:11:00.541526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.541340    4531 generic.go:356] "Generic (PLEG): container finished" podID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerID="a46911c5d43a04bd1d33c9b1a146cfe2ea65482868e63527c6d247964e5176ab" exitCode=0
Nov 03 01:11:00.541526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.541410    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" event={"ID":"31eda088-9b56-4d3f-8ef9-9b9b21f2989e","Type":"ContainerDied","Data":"a46911c5d43a04bd1d33c9b1a146cfe2ea65482868e63527c6d247964e5176ab"}
Nov 03 01:11:00.689914 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.689592    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 01:11:00.863598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.863563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.863809 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.863659    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.863809 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.863725    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.863809 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.863782    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.864025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.863821    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.864081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864020    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.864339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864115    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:00.864445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864348    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:00.864445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864356    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:00.864445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864380    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:00.864618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864491    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.864618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864544    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.864735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.864719    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\") pod \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\" (UID: \"31eda088-9b56-4d3f-8ef9-9b9b21f2989e\") "
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865142    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865596    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865617    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865633    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865647    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.865715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.865662    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.866425 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.866374    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:00.866675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.866647    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:00.866865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.866841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "kube-api-access-k9v5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:00.867023 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.867007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo" pod "31eda088-9b56-4d3f-8ef9-9b9b21f2989e" (UID: "31eda088-9b56-4d3f-8ef9-9b9b21f2989e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:11:00.967341 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.967301    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9v5p\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-kube-api-access-k9v5p\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.967341 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.967337    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.967591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.967354    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:00.967591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:00.967368    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31eda088-9b56-4d3f-8ef9-9b9b21f2989e-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:11:01.552813 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:01.552786    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"
Nov 03 01:11:01.553344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:01.552830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q" event={"ID":"31eda088-9b56-4d3f-8ef9-9b9b21f2989e","Type":"ContainerDied","Data":"03e424cc510ef63fbeebf39f15a72263758ce751645caf2406ae4b4f108bd9c4"}
Nov 03 01:11:01.553344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:01.552906    4531 scope.go:122] "RemoveContainer" containerID="a46911c5d43a04bd1d33c9b1a146cfe2ea65482868e63527c6d247964e5176ab"
Nov 03 01:11:01.577051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:01.577022    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 01:11:01.581614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:01.581590    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-t9s6q"]
Nov 03 01:11:02.915911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:02.915861    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" path="/var/lib/kubelet/pods/31eda088-9b56-4d3f-8ef9-9b9b21f2989e/volumes"
Nov 03 01:11:09.887272 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:11:09.887197    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:11:27.229481 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.227957    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:11:27.233191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.232936    4531 cpu_manager.go:414] "containerMap: removing container" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy"
Nov 03 01:11:27.233191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.232965    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy"
Nov 03 01:11:27.233191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.233015    4531 cpu_manager.go:414] "containerMap: removing container" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" containerName="test-curl-7034-aaa247"
Nov 03 01:11:27.233191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.233027    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" containerName="test-curl-7034-aaa247"
Nov 03 01:11:27.233549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.233500    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="31eda088-9b56-4d3f-8ef9-9b9b21f2989e" containerName="istio-proxy"
Nov 03 01:11:27.233549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.233520    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f47cad-ca64-4a68-8e8a-78ebaa856335" containerName="test-curl-7034-aaa247"
Nov 03 01:11:27.250131 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.247528    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.253731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.253697    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3c5983-serving-cert\""
Nov 03 01:11:27.265664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.265632    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:11:27.351820 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.351779    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zz9n\" (UniqueName: \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.352093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.352045    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.352241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.352128    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.352439 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.352421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.456224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.456151    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zz9n\" (UniqueName: \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.456505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.456334    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.456505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.456378    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.456708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.456612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.457991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.457603    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.464348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.462287    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.464348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.462737    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.494001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.492811    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zz9n\" (UniqueName: \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\") pod \"maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.592289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.592250    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:27.888423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:27.888388    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:11:27.890583 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:11:27.890551    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a6a44e_0da1_4400_80cf_62dc78e274e1.slice/crio-f018bfb63da55ae3451e990fef6dfc8a62947eeeb73438981c84ae39f0a3d3cd WatchSource:0}: Error finding container f018bfb63da55ae3451e990fef6dfc8a62947eeeb73438981c84ae39f0a3d3cd: Status 404 returned error can't find the container with id f018bfb63da55ae3451e990fef6dfc8a62947eeeb73438981c84ae39f0a3d3cd
Nov 03 01:11:28.341876 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:28.341834    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" event={"ID":"d4a6a44e-0da1-4400-80cf-62dc78e274e1","Type":"ContainerStarted","Data":"f018bfb63da55ae3451e990fef6dfc8a62947eeeb73438981c84ae39f0a3d3cd"}
Nov 03 01:11:29.353561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:29.353524    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" event={"ID":"d4a6a44e-0da1-4400-80cf-62dc78e274e1","Type":"ContainerStarted","Data":"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"}
Nov 03 01:11:29.354000 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:29.353646    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:29.375859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:29.375789    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" podStartSLOduration=1.891222642 podStartE2EDuration="2.375775126s" totalImagesPullingTime="484.552484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:29.369488943 +0000 UTC m=+1834.583232615" watchObservedRunningTime="2026-11-03 01:11:29.375775126 +0000 UTC m=+1834.589518797"
Nov 03 01:11:35.370912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:35.370879    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:11:36.656319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.656223    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:11:36.663298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.663270    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:11:36.663797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.663689    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.677433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.677392    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-57079b-serving-cert\""
Nov 03 01:11:36.758870 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.758836    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwx8\" (UniqueName: \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.759077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.758975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.759151 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.759126    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.759213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.759189    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.880184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.863292    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.891792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.891749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.891995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.891909    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwx8\" (UniqueName: \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.892262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.892036    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.893241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.865239    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.902759 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.902626    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.904661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.904630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.920668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.920578    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwx8\" (UniqueName: \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\") pod \"maas-api-e2e-extmod-b-57079b-946cf9869-thjcr\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:36.987466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:36.987424    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:37.313408 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:11:37.304275    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00ee078_1c60_43a7_bed4_61e7d7d1fdaa.slice/crio-46d2794c218d6088d14456caf425acc433db6a75cbba826b470a6016fe065f29 WatchSource:0}: Error finding container 46d2794c218d6088d14456caf425acc433db6a75cbba826b470a6016fe065f29: Status 404 returned error can't find the container with id 46d2794c218d6088d14456caf425acc433db6a75cbba826b470a6016fe065f29
Nov 03 01:11:37.313408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:37.306422    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:11:37.505184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:37.504133    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" event={"ID":"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa","Type":"ContainerStarted","Data":"46d2794c218d6088d14456caf425acc433db6a75cbba826b470a6016fe065f29"}
Nov 03 01:11:38.513991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:38.513952    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" event={"ID":"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa","Type":"ContainerStarted","Data":"4ac5bdc1bb1f1293c7cb044add1e366fe3294606435eec5d9d954c5194a4d8e7"}
Nov 03 01:11:38.514448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:38.514036    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:11:38.537433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:38.537370    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" podStartSLOduration=2.083459512 podStartE2EDuration="2.537349737s" totalImagesPullingTime="453.890225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:38.532431585 +0000 UTC m=+1843.746175258" watchObservedRunningTime="2026-11-03 01:11:38.537349737 +0000 UTC m=+1843.751093409"
Nov 03 01:11:39.879696 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:11:39.879659    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:11:44.529066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:11:44.529031    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:12:09.879506 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:12:09.879466    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:12:39.880009 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:12:39.879968    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:13:09.883092 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:13:09.881648    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:13:39.881130 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:13:39.881082    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:14:09.879076 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:14:09.879039    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:14:39.879400 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:14:39.879318    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:15:00.188641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.188597    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:00.195461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.195430    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.198421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.196021    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:00.207638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.206850    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.220270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.211748    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:00.220270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.216693    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:00.233227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.233193    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:00.238922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.238885    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.269486 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.269454    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\""
Nov 03 01:15:00.272399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.270653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2kd\" (UniqueName: \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.272525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.272478    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.272839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.272811    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.273076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.273060    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqqg\" (UniqueName: \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.280540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.280493    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:00.286738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.286708    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.289359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.289332    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:00.294771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.294742    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:00.375082 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4qd\" (UniqueName: \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.375326 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375087    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfmj\" (UniqueName: \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.375326 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375242    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqqg\" (UniqueName: \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.375461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.375461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2kd\" (UniqueName: \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.375461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375451    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.375604 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375482    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.375824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.375797    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.376392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.376363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.376588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.376570    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.384592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.384566    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2kd\" (UniqueName: \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\") pod \"maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.384746 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.384656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqqg\" (UniqueName: \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.491248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.488417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.491248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.488528    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.491248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.490798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4qd\" (UniqueName: \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.491248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.490845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfmj\" (UniqueName: \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.492769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.492706    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.496748 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.496686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.519885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.514034    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4qd\" (UniqueName: \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.535183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.525123    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:00.541740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.541702    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfmj\" (UniqueName: \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.590855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.590814    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:00.602508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.602470    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:00.611617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.611582    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:00.838540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.838494    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:00.853463 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:00.853409    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2764d1af_7e0f_40d0_82e3_5e4c37ad79c5.slice/crio-a5e433d96c2ceac3f825979f952cdbdca8ee2b5c54094c18252dade90e2ada1a WatchSource:0}: Error finding container a5e433d96c2ceac3f825979f952cdbdca8ee2b5c54094c18252dade90e2ada1a: Status 404 returned error can't find the container with id a5e433d96c2ceac3f825979f952cdbdca8ee2b5c54094c18252dade90e2ada1a
Nov 03 01:15:00.919171 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.919130    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:00.920881 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:00.920848    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329aaa04_706d_4222_8dc9_0641446dc77c.slice/crio-28048c04602e04cc8376f46e828a99cdccc295705b7baddda719af3d059f7bd1 WatchSource:0}: Error finding container 28048c04602e04cc8376f46e828a99cdccc295705b7baddda719af3d059f7bd1: Status 404 returned error can't find the container with id 28048c04602e04cc8376f46e828a99cdccc295705b7baddda719af3d059f7bd1
Nov 03 01:15:00.974959 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.974930    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:00.975185 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:00.975117    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5555c5e8_a9b3_42af_9167_95eabf7e899e.slice/crio-4aea5bf9349552d5835d179e3ec9792a7af2c54f3693bba29fc7ac46ba1609bb WatchSource:0}: Error finding container 4aea5bf9349552d5835d179e3ec9792a7af2c54f3693bba29fc7ac46ba1609bb: Status 404 returned error can't find the container with id 4aea5bf9349552d5835d179e3ec9792a7af2c54f3693bba29fc7ac46ba1609bb
Nov 03 01:15:00.980706 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:00.980675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:01.061242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:01.060371    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerStarted","Data":"28048c04602e04cc8376f46e828a99cdccc295705b7baddda719af3d059f7bd1"}
Nov 03 01:15:01.062086 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:01.062055    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerStarted","Data":"4aea5bf9349552d5835d179e3ec9792a7af2c54f3693bba29fc7ac46ba1609bb"}
Nov 03 01:15:01.063648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:01.063623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerStarted","Data":"14851f8810c041dfb995266463b79cd0e12b15b135228de484c2d65972f24230"}
Nov 03 01:15:01.065092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:01.065066    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerStarted","Data":"a5e433d96c2ceac3f825979f952cdbdca8ee2b5c54094c18252dade90e2ada1a"}
Nov 03 01:15:09.967255 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:09.967211    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:15:10.269902 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.268047    4531 generic.go:356] "Generic (PLEG): container finished" podID="329aaa04-706d-4222-8dc9-0641446dc77c" containerID="3e45720b326ecdb5c9933e982a9c95f28dd436cf851d32cd08bd3a9c1089147b" exitCode=60
Nov 03 01:15:10.269902 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.268834    4531 scope.go:122] "RemoveContainer" containerID="3e45720b326ecdb5c9933e982a9c95f28dd436cf851d32cd08bd3a9c1089147b"
Nov 03 01:15:10.269902 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.269410    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerDied","Data":"3e45720b326ecdb5c9933e982a9c95f28dd436cf851d32cd08bd3a9c1089147b"}
Nov 03 01:15:10.357808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.357768    4531 generic.go:356] "Generic (PLEG): container finished" podID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerID="0e765f1fd822c12cdee30b5d4efdaad91ff3a7400733c24de254f231d627b390" exitCode=60
Nov 03 01:15:10.358285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.358239    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerDied","Data":"0e765f1fd822c12cdee30b5d4efdaad91ff3a7400733c24de254f231d627b390"}
Nov 03 01:15:10.383596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:10.383558    4531 scope.go:122] "RemoveContainer" containerID="0e765f1fd822c12cdee30b5d4efdaad91ff3a7400733c24de254f231d627b390"
Nov 03 01:15:11.369735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:11.369700    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerStarted","Data":"d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"}
Nov 03 01:15:11.372070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:11.372035    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerStarted","Data":"07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"}
Nov 03 01:15:11.387262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:11.387213    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" podStartSLOduration=3.476711041 podStartE2EDuration="11.387198585s" totalImagesPullingTime="7.910487544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:11.385289935 +0000 UTC m=+2056.599033609" watchObservedRunningTime="2026-11-03 01:15:11.387198585 +0000 UTC m=+2056.600942277"
Nov 03 01:15:11.408654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:11.408606    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" podStartSLOduration=3.4418991119999998 podStartE2EDuration="11.408591451s" totalImagesPullingTime="7.966692339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:11.406907939 +0000 UTC m=+2056.620651608" watchObservedRunningTime="2026-11-03 01:15:11.408591451 +0000 UTC m=+2056.622335123"
Nov 03 01:15:12.382712 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.382673    4531 generic.go:356] "Generic (PLEG): container finished" podID="329aaa04-706d-4222-8dc9-0641446dc77c" containerID="07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50" exitCode=60
Nov 03 01:15:12.383304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.382731    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerDied","Data":"07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"}
Nov 03 01:15:12.383304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.382814    4531 scope.go:122] "RemoveContainer" containerID="3e45720b326ecdb5c9933e982a9c95f28dd436cf851d32cd08bd3a9c1089147b"
Nov 03 01:15:12.383304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.383267    4531 scope.go:122] "RemoveContainer" containerID="07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"
Nov 03 01:15:12.383723 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:12.383693    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv_odh-ai-gateway-infra(329aaa04-706d-4222-8dc9-0641446dc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" podUID="329aaa04-706d-4222-8dc9-0641446dc77c"
Nov 03 01:15:12.386653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.386619    4531 generic.go:356] "Generic (PLEG): container finished" podID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerID="d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531" exitCode=60
Nov 03 01:15:12.386765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.386667    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerDied","Data":"d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"}
Nov 03 01:15:12.387067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.387044    4531 scope.go:122] "RemoveContainer" containerID="d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"
Nov 03 01:15:12.387507 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:12.387475    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2_odh-ai-gateway-infra(5555c5e8-a9b3-42af-9167-95eabf7e899e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e"
Nov 03 01:15:12.399263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:12.399237    4531 scope.go:122] "RemoveContainer" containerID="0e765f1fd822c12cdee30b5d4efdaad91ff3a7400733c24de254f231d627b390"
Nov 03 01:15:13.396515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:13.396486    4531 scope.go:122] "RemoveContainer" containerID="07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"
Nov 03 01:15:13.396909 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:13.396880    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv_odh-ai-gateway-infra(329aaa04-706d-4222-8dc9-0641446dc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" podUID="329aaa04-706d-4222-8dc9-0641446dc77c"
Nov 03 01:15:13.398299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:13.398271    4531 scope.go:122] "RemoveContainer" containerID="d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"
Nov 03 01:15:13.398669 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:13.398642    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2_odh-ai-gateway-infra(5555c5e8-a9b3-42af-9167-95eabf7e899e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e"
Nov 03 01:15:13.399735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:13.399711    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerStarted","Data":"4e583f5b43335f0680614d507381fc4322d21d07290af9f71d8978d3bbf589c4"}
Nov 03 01:15:13.517242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:13.517062    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" podStartSLOduration=1.51381761 podStartE2EDuration="13.517041212s" totalImagesPullingTime="12.003223602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:13.511338632 +0000 UTC m=+2058.725082304" watchObservedRunningTime="2026-11-03 01:15:13.517041212 +0000 UTC m=+2058.730784886"
Nov 03 01:15:14.456470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:14.450480    4531 generic.go:356] "Generic (PLEG): container finished" podID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerID="4e583f5b43335f0680614d507381fc4322d21d07290af9f71d8978d3bbf589c4" exitCode=60
Nov 03 01:15:14.456470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:14.450644    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerDied","Data":"4e583f5b43335f0680614d507381fc4322d21d07290af9f71d8978d3bbf589c4"}
Nov 03 01:15:14.456470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:14.451154    4531 scope.go:122] "RemoveContainer" containerID="4e583f5b43335f0680614d507381fc4322d21d07290af9f71d8978d3bbf589c4"
Nov 03 01:15:16.490150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.490055    4531 generic.go:356] "Generic (PLEG): container finished" podID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerID="3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742" exitCode=60
Nov 03 01:15:16.490150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.490137    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerDied","Data":"3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742"}
Nov 03 01:15:16.490736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.490209    4531 scope.go:122] "RemoveContainer" containerID="4e583f5b43335f0680614d507381fc4322d21d07290af9f71d8978d3bbf589c4"
Nov 03 01:15:16.490736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.490587    4531 scope.go:122] "RemoveContainer" containerID="3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742"
Nov 03 01:15:16.491066 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:16.491034    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6_odh-ai-gateway-infra(2764d1af-7e0f-40d0-82e3-5e4c37ad79c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5"
Nov 03 01:15:16.492497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.492438    4531 generic.go:356] "Generic (PLEG): container finished" podID="17cc790e-ec67-48fb-b4e5-583caf820577" containerID="3ba9aa5cfe7158081980b94cd600a9184fa6c11c122735cfe034fe12bfd72f5b" exitCode=60
Nov 03 01:15:16.492497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.492489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerDied","Data":"3ba9aa5cfe7158081980b94cd600a9184fa6c11c122735cfe034fe12bfd72f5b"}
Nov 03 01:15:16.492801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:16.492787    4531 scope.go:122] "RemoveContainer" containerID="3ba9aa5cfe7158081980b94cd600a9184fa6c11c122735cfe034fe12bfd72f5b"
Nov 03 01:15:17.503331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:17.503292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerStarted","Data":"8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"}
Nov 03 01:15:17.505290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:17.505267    4531 scope.go:122] "RemoveContainer" containerID="3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742"
Nov 03 01:15:17.505617 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:17.505598    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6_odh-ai-gateway-infra(2764d1af-7e0f-40d0-82e3-5e4c37ad79c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5"
Nov 03 01:15:17.523821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:17.523769    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" podStartSLOduration=3.294273493 podStartE2EDuration="17.523754579s" totalImagesPullingTime="14.229481086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:17.518088157 +0000 UTC m=+2062.731831829" watchObservedRunningTime="2026-11-03 01:15:17.523754579 +0000 UTC m=+2062.737498250"
Nov 03 01:15:18.515479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:18.515444    4531 generic.go:356] "Generic (PLEG): container finished" podID="17cc790e-ec67-48fb-b4e5-583caf820577" containerID="8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710" exitCode=60
Nov 03 01:15:18.515950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:18.515532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerDied","Data":"8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"}
Nov 03 01:15:18.515950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:18.515595    4531 scope.go:122] "RemoveContainer" containerID="3ba9aa5cfe7158081980b94cd600a9184fa6c11c122735cfe034fe12bfd72f5b"
Nov 03 01:15:18.515950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:18.515895    4531 scope.go:122] "RemoveContainer" containerID="8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"
Nov 03 01:15:18.516304 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:18.516264    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q_odh-ai-gateway-infra(17cc790e-ec67-48fb-b4e5-583caf820577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" podUID="17cc790e-ec67-48fb-b4e5-583caf820577"
Nov 03 01:15:19.526825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:19.526786    4531 scope.go:122] "RemoveContainer" containerID="8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"
Nov 03 01:15:19.527371 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:19.527247    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q_odh-ai-gateway-infra(17cc790e-ec67-48fb-b4e5-583caf820577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" podUID="17cc790e-ec67-48fb-b4e5-583caf820577"
Nov 03 01:15:22.896674 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:22.896631    4531 scope.go:122] "RemoveContainer" containerID="07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"
Nov 03 01:15:23.584763 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:23.584727    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerStarted","Data":"8ddd570e1967684b02eb580376075d71f3f8f7a7aee75508172b028d514df1ad"}
Nov 03 01:15:24.599717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:24.599678    4531 generic.go:356] "Generic (PLEG): container finished" podID="329aaa04-706d-4222-8dc9-0641446dc77c" containerID="8ddd570e1967684b02eb580376075d71f3f8f7a7aee75508172b028d514df1ad" exitCode=60
Nov 03 01:15:24.600205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:24.599748    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerDied","Data":"8ddd570e1967684b02eb580376075d71f3f8f7a7aee75508172b028d514df1ad"}
Nov 03 01:15:24.600205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:24.599820    4531 scope.go:122] "RemoveContainer" containerID="07d66d54c63a7cdbe7625bbdcadf9cda08f0dc39f514d558f83307317ad3dd50"
Nov 03 01:15:24.600361 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:24.600327    4531 scope.go:122] "RemoveContainer" containerID="8ddd570e1967684b02eb580376075d71f3f8f7a7aee75508172b028d514df1ad"
Nov 03 01:15:24.600842 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:24.600813    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv_odh-ai-gateway-infra(329aaa04-706d-4222-8dc9-0641446dc77c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" podUID="329aaa04-706d-4222-8dc9-0641446dc77c"
Nov 03 01:15:24.702812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:24.701075    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:25.822408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.822370    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:25.853957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.853913    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:25.855423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855396    4531 cpu_manager.go:414] "containerMap: removing container" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.855605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855582    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.855719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855690    4531 cpu_manager.go:414] "containerMap: removing container" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.855719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855701    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.855888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855876    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.855923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.855893    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:25.865443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.865368    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:25.868083 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.868054    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:25.896898 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.896870    4531 scope.go:122] "RemoveContainer" containerID="d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"
Nov 03 01:15:25.917534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.917500    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\") pod \"329aaa04-706d-4222-8dc9-0641446dc77c\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") "
Nov 03 01:15:25.917936 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.917910    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\" (UniqueName: \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\") pod \"329aaa04-706d-4222-8dc9-0641446dc77c\" (UID: \"329aaa04-706d-4222-8dc9-0641446dc77c\") "
Nov 03 01:15:25.918292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.917899    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca" pod "329aaa04-706d-4222-8dc9-0641446dc77c" (UID: "329aaa04-706d-4222-8dc9-0641446dc77c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:25.918692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.918667    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/329aaa04-706d-4222-8dc9-0641446dc77c-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:25.922469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:25.922435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg" pod "329aaa04-706d-4222-8dc9-0641446dc77c" (UID: "329aaa04-706d-4222-8dc9-0641446dc77c"). InnerVolumeSpecName "kube-api-access-lxqqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:26.020653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.020598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.020922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.020854    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mw4z\" (UniqueName: \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.021039 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.021012    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqqg\" (UniqueName: \"kubernetes.io/projected/329aaa04-706d-4222-8dc9-0641446dc77c-kube-api-access-lxqqg\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:26.122133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.122039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.122347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.122227    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mw4z\" (UniqueName: \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.122795 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.122776    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.131339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.131306    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mw4z\" (UniqueName: \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\") pod \"maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.186315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.186271    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:26.483695 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:26.483660    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353e8d9e_eb46_480c_b3d2_715ab122093c.slice/crio-0f12bf4cc749449dce22e4b8d784beaabff4b0054f6034b8012d2ef42dd8c65e WatchSource:0}: Error finding container 0f12bf4cc749449dce22e4b8d784beaabff4b0054f6034b8012d2ef42dd8c65e: Status 404 returned error can't find the container with id 0f12bf4cc749449dce22e4b8d784beaabff4b0054f6034b8012d2ef42dd8c65e
Nov 03 01:15:26.484991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.484964    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:26.619763 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.619723    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerStarted","Data":"0f12bf4cc749449dce22e4b8d784beaabff4b0054f6034b8012d2ef42dd8c65e"}
Nov 03 01:15:26.621422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.621391    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"
Nov 03 01:15:26.621568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.621397    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv" event={"ID":"329aaa04-706d-4222-8dc9-0641446dc77c","Type":"ContainerDied","Data":"28048c04602e04cc8376f46e828a99cdccc295705b7baddda719af3d059f7bd1"}
Nov 03 01:15:26.621568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.621543    4531 scope.go:122] "RemoveContainer" containerID="8ddd570e1967684b02eb580376075d71f3f8f7a7aee75508172b028d514df1ad"
Nov 03 01:15:26.623786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.623760    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerStarted","Data":"e2064f1eaab10270b7094a8460ea97b1062ec360e11a4b9e4773b6eb2603ef79"}
Nov 03 01:15:26.668056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.668006    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:26.673430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.673391    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894475-x7lgv"]
Nov 03 01:15:26.911257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:26.911217    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" path="/var/lib/kubelet/pods/329aaa04-706d-4222-8dc9-0641446dc77c/volumes"
Nov 03 01:15:27.635555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.635521    4531 generic.go:356] "Generic (PLEG): container finished" podID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerID="e2064f1eaab10270b7094a8460ea97b1062ec360e11a4b9e4773b6eb2603ef79" exitCode=60
Nov 03 01:15:27.635753 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.635603    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerDied","Data":"e2064f1eaab10270b7094a8460ea97b1062ec360e11a4b9e4773b6eb2603ef79"}
Nov 03 01:15:27.635753 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.635663    4531 scope.go:122] "RemoveContainer" containerID="d73099ee4316429f5ef50b4eea505ad4381c207b89ee258832650b23f4840531"
Nov 03 01:15:27.636152 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.636133    4531 scope.go:122] "RemoveContainer" containerID="e2064f1eaab10270b7094a8460ea97b1062ec360e11a4b9e4773b6eb2603ef79"
Nov 03 01:15:27.636987 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:27.636629    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2_odh-ai-gateway-infra(5555c5e8-a9b3-42af-9167-95eabf7e899e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e"
Nov 03 01:15:27.637625 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.637605    4531 generic.go:356] "Generic (PLEG): container finished" podID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerID="7ca495ad0746b569a7482f8565ff96379368ad9c9d90075824f9a510dc41fcc6" exitCode=60
Nov 03 01:15:27.637731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.637682    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerDied","Data":"7ca495ad0746b569a7482f8565ff96379368ad9c9d90075824f9a510dc41fcc6"}
Nov 03 01:15:27.638004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.637987    4531 scope.go:122] "RemoveContainer" containerID="7ca495ad0746b569a7482f8565ff96379368ad9c9d90075824f9a510dc41fcc6"
Nov 03 01:15:27.671746 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.671712    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:27.895940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:27.895911    4531 scope.go:122] "RemoveContainer" containerID="3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742"
Nov 03 01:15:28.657995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.657944    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerStarted","Data":"e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"}
Nov 03 01:15:28.660957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.660927    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerStarted","Data":"881db96bfc10df2418b2e2c4b3bf48e5ce000445ffcf0194e06c2a4d811d0e33"}
Nov 03 01:15:28.710801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.710745    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" podStartSLOduration=28.710729112 podStartE2EDuration="28.710729112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:28.706663242 +0000 UTC m=+2073.920406937" watchObservedRunningTime="2026-11-03 01:15:28.710729112 +0000 UTC m=+2073.924472783"
Nov 03 01:15:28.835364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.835325    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:28.870437 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.870387    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:28.871191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871155    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871189    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871212    4531 cpu_manager.go:414] "containerMap: removing container" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:28.871382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871220    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:28.871382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871268    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871273    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871394    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871407    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.871596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871414    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="329aaa04-706d-4222-8dc9-0641446dc77c" containerName="cleanup"
Nov 03 01:15:28.871596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.871421    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:28.875450 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.875414    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:28.886497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.886466    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:28.940094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.939995    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\" (UniqueName: \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\") pod \"5555c5e8-a9b3-42af-9167-95eabf7e899e\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") "
Nov 03 01:15:28.940356 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.940332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\") pod \"5555c5e8-a9b3-42af-9167-95eabf7e899e\" (UID: \"5555c5e8-a9b3-42af-9167-95eabf7e899e\") "
Nov 03 01:15:28.940766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.940729    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca" pod "5555c5e8-a9b3-42af-9167-95eabf7e899e" (UID: "5555c5e8-a9b3-42af-9167-95eabf7e899e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:28.941123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.941101    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555c5e8-a9b3-42af-9167-95eabf7e899e-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:28.942571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.942544    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj" pod "5555c5e8-a9b3-42af-9167-95eabf7e899e" (UID: "5555c5e8-a9b3-42af-9167-95eabf7e899e"). InnerVolumeSpecName "kube-api-access-wxfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:28.988597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.988354    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 01:15:28.988797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:28.988711    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy" containerID="cri-o://343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518" gracePeriod=30
Nov 03 01:15:29.042558 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.042513    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4dl\" (UniqueName: \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.043106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.042738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.043106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.043067    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfmj\" (UniqueName: \"kubernetes.io/projected/5555c5e8-a9b3-42af-9167-95eabf7e899e-kube-api-access-wxfmj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:29.145027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.144992    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4dl\" (UniqueName: \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.145209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.145188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.145798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.145777    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.153260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.153235    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4dl\" (UniqueName: \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\") pod \"maas-api-key-cleanup-29894475-qrpq7\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.209021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.208939    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:29.671602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.671573    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"
Nov 03 01:15:29.671602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.671592    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2" event={"ID":"5555c5e8-a9b3-42af-9167-95eabf7e899e","Type":"ContainerDied","Data":"4aea5bf9349552d5835d179e3ec9792a7af2c54f3693bba29fc7ac46ba1609bb"}
Nov 03 01:15:29.672210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.671755    4531 scope.go:122] "RemoveContainer" containerID="e2064f1eaab10270b7094a8460ea97b1062ec360e11a4b9e4773b6eb2603ef79"
Nov 03 01:15:29.674905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.674877    4531 generic.go:356] "Generic (PLEG): container finished" podID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerID="e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad" exitCode=60
Nov 03 01:15:29.675061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.675030    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerDied","Data":"e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"}
Nov 03 01:15:29.675401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.675387    4531 scope.go:122] "RemoveContainer" containerID="e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"
Nov 03 01:15:29.675832 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:29.675780    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp_odh-ai-gateway-infra(353e8d9e-eb46-480c-b3d2-715ab122093c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c"
Nov 03 01:15:29.679596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.679579    4531 generic.go:356] "Generic (PLEG): container finished" podID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerID="881db96bfc10df2418b2e2c4b3bf48e5ce000445ffcf0194e06c2a4d811d0e33" exitCode=60
Nov 03 01:15:29.679732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.679614    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerDied","Data":"881db96bfc10df2418b2e2c4b3bf48e5ce000445ffcf0194e06c2a4d811d0e33"}
Nov 03 01:15:29.680129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.680112    4531 scope.go:122] "RemoveContainer" containerID="881db96bfc10df2418b2e2c4b3bf48e5ce000445ffcf0194e06c2a4d811d0e33"
Nov 03 01:15:29.680484 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:29.680459    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6_odh-ai-gateway-infra(2764d1af-7e0f-40d0-82e3-5e4c37ad79c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5"
Nov 03 01:15:29.692633 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.692443    4531 scope.go:122] "RemoveContainer" containerID="7ca495ad0746b569a7482f8565ff96379368ad9c9d90075824f9a510dc41fcc6"
Nov 03 01:15:29.715616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.715429    4531 scope.go:122] "RemoveContainer" containerID="3f3a90d30b4d96cc017b230cb413ec712372a70b8285927dbd9d431bf74fc742"
Nov 03 01:15:29.715616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.715537    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:29.717587 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:29.717547    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a45e1bd_4889_404f_9b3b_4a52cddaec90.slice/crio-a201410afff2c7baf18e9f178e5ac1fe5e7c7f59959131c83f57b24736ad2e58 WatchSource:0}: Error finding container a201410afff2c7baf18e9f178e5ac1fe5e7c7f59959131c83f57b24736ad2e58: Status 404 returned error can't find the container with id a201410afff2c7baf18e9f178e5ac1fe5e7c7f59959131c83f57b24736ad2e58
Nov 03 01:15:29.721102 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.721063    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:29.727005 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.726982    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:29.739389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:29.739357    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894475-5xpg2"]
Nov 03 01:15:30.690707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.690674    4531 scope.go:122] "RemoveContainer" containerID="e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"
Nov 03 01:15:30.691229 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:30.691129    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp_odh-ai-gateway-infra(353e8d9e-eb46-480c-b3d2-715ab122093c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c"
Nov 03 01:15:30.694119 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.694086    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerStarted","Data":"658e79aa651bccc0b982a8402991aac0970ebbf4ee312f783c4303da99f59ce1"}
Nov 03 01:15:30.694285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.694136    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerStarted","Data":"a201410afff2c7baf18e9f178e5ac1fe5e7c7f59959131c83f57b24736ad2e58"}
Nov 03 01:15:30.748392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.748330    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" podStartSLOduration=30.748314593 podStartE2EDuration="30.748314593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:30.742396602 +0000 UTC m=+2075.956140274" watchObservedRunningTime="2026-11-03 01:15:30.748314593 +0000 UTC m=+2075.962058264"
Nov 03 01:15:30.906549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.906513    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" path="/var/lib/kubelet/pods/5555c5e8-a9b3-42af-9167-95eabf7e899e/volumes"
Nov 03 01:15:30.942641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.942572    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:30.984486 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.984360    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:30.985835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985797    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.985835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985831    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.986020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985908    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:30.986020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985916    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555c5e8-a9b3-42af-9167-95eabf7e899e" containerName="cleanup"
Nov 03 01:15:30.986020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985939    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.986020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.985947    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.986255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.986120    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.986255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.986139    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.986255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.986173    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:30.990728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.990701    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:30.997430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:30.997398    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:31.090090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.090044    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\" (UniqueName: \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\") pod \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") "
Nov 03 01:15:31.090390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.090142    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\") pod \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\" (UID: \"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5\") "
Nov 03 01:15:31.094191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.090899    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.094191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.090977    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7q2\" (UniqueName: \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.113188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.109619    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca" pod "2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" (UID: "2764d1af-7e0f-40d0-82e3-5e4c37ad79c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:31.123412 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.123365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd" pod "2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" (UID: "2764d1af-7e0f-40d0-82e3-5e4c37ad79c5"). InnerVolumeSpecName "kube-api-access-vr2kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:31.192849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.192748    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.192849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.192804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7q2\" (UniqueName: \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.193092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.193001    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr2kd\" (UniqueName: \"kubernetes.io/projected/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-kube-api-access-vr2kd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:31.193092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.193012    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:31.193416 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.193394    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.201721 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.201688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7q2\" (UniqueName: \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\") pod \"maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.309308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.309273    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:31.602565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.602529    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:31.604565 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:31.604537    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cece9b7_6774_4168_aea5_0a86688d01cd.slice/crio-bd96cc058accd7a6934b0bf0b9d32301423c0e61df22dacf7f733afc59ef4f7e WatchSource:0}: Error finding container bd96cc058accd7a6934b0bf0b9d32301423c0e61df22dacf7f733afc59ef4f7e: Status 404 returned error can't find the container with id bd96cc058accd7a6934b0bf0b9d32301423c0e61df22dacf7f733afc59ef4f7e
Nov 03 01:15:31.763691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.754680    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6" event={"ID":"2764d1af-7e0f-40d0-82e3-5e4c37ad79c5","Type":"ContainerDied","Data":"a5e433d96c2ceac3f825979f952cdbdca8ee2b5c54094c18252dade90e2ada1a"}
Nov 03 01:15:31.763691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.754762    4531 scope.go:122] "RemoveContainer" containerID="881db96bfc10df2418b2e2c4b3bf48e5ce000445ffcf0194e06c2a4d811d0e33"
Nov 03 01:15:31.763691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.756076    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"
Nov 03 01:15:31.768103 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.768034    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerStarted","Data":"bd96cc058accd7a6934b0bf0b9d32301423c0e61df22dacf7f733afc59ef4f7e"}
Nov 03 01:15:31.778256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.771398    4531 generic.go:356] "Generic (PLEG): container finished" podID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerID="658e79aa651bccc0b982a8402991aac0970ebbf4ee312f783c4303da99f59ce1" exitCode=60
Nov 03 01:15:31.778256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.771624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerDied","Data":"658e79aa651bccc0b982a8402991aac0970ebbf4ee312f783c4303da99f59ce1"}
Nov 03 01:15:31.778256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.772307    4531 scope.go:122] "RemoveContainer" containerID="658e79aa651bccc0b982a8402991aac0970ebbf4ee312f783c4303da99f59ce1"
Nov 03 01:15:31.817897 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.817863    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:31.823394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.823360    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-7c759a-29894475-6qwm6"]
Nov 03 01:15:31.896599 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:31.896505    4531 scope.go:122] "RemoveContainer" containerID="8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"
Nov 03 01:15:32.784002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.783960    4531 generic.go:356] "Generic (PLEG): container finished" podID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerID="e45e82c0245362cb26c30a89f570a2880755b6961e273d699c1cd4336c2c2721" exitCode=60
Nov 03 01:15:32.784437 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.784002    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerDied","Data":"e45e82c0245362cb26c30a89f570a2880755b6961e273d699c1cd4336c2c2721"}
Nov 03 01:15:32.784512 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.784456    4531 scope.go:122] "RemoveContainer" containerID="e45e82c0245362cb26c30a89f570a2880755b6961e273d699c1cd4336c2c2721"
Nov 03 01:15:32.786210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.786187    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerStarted","Data":"5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"}
Nov 03 01:15:32.788744 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.788641    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerStarted","Data":"22c8179bc72bd972898287a4fc17671e2ae59528fdbbbd3808c94bb42ea7175f"}
Nov 03 01:15:32.905270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:32.905236    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" path="/var/lib/kubelet/pods/2764d1af-7e0f-40d0-82e3-5e4c37ad79c5/volumes"
Nov 03 01:15:33.798432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.798398    4531 generic.go:356] "Generic (PLEG): container finished" podID="17cc790e-ec67-48fb-b4e5-583caf820577" containerID="22c8179bc72bd972898287a4fc17671e2ae59528fdbbbd3808c94bb42ea7175f" exitCode=60
Nov 03 01:15:33.798925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.798472    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerDied","Data":"22c8179bc72bd972898287a4fc17671e2ae59528fdbbbd3808c94bb42ea7175f"}
Nov 03 01:15:33.798925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.798548    4531 scope.go:122] "RemoveContainer" containerID="8cd644bf9042d62a07e3e43e9b9b4f23615fc0220327e23514b2350e664fe710"
Nov 03 01:15:33.799061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.799022    4531 scope.go:122] "RemoveContainer" containerID="22c8179bc72bd972898287a4fc17671e2ae59528fdbbbd3808c94bb42ea7175f"
Nov 03 01:15:33.799507 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:33.799472    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q_odh-ai-gateway-infra(17cc790e-ec67-48fb-b4e5-583caf820577)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" podUID="17cc790e-ec67-48fb-b4e5-583caf820577"
Nov 03 01:15:33.800608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.800583    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerStarted","Data":"89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"}
Nov 03 01:15:33.803369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.803334    4531 generic.go:356] "Generic (PLEG): container finished" podID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerID="5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a" exitCode=60
Nov 03 01:15:33.803459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.803391    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerDied","Data":"5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"}
Nov 03 01:15:33.803698 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.803684    4531 scope.go:122] "RemoveContainer" containerID="5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"
Nov 03 01:15:33.804058 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:33.804042    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-qrpq7_odh-ai-gateway-infra(2a45e1bd-4889-404f-9b3b-4a52cddaec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90"
Nov 03 01:15:33.813052 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.813030    4531 scope.go:122] "RemoveContainer" containerID="658e79aa651bccc0b982a8402991aac0970ebbf4ee312f783c4303da99f59ce1"
Nov 03 01:15:33.832564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.832534    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:33.865105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:33.865052    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" podStartSLOduration=33.865037607 podStartE2EDuration="33.865037607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:33.853238268 +0000 UTC m=+2079.066981965" watchObservedRunningTime="2026-11-03 01:15:33.865037607 +0000 UTC m=+2079.078781278"
Nov 03 01:15:34.323854 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.323823    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 01:15:34.371541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371403    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371468    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371524    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371790 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371597    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371812    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371864 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371849    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.371975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.371955    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.372070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.372029    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.372070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.372059    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\") pod \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\" (UID: \"0e582b53-d78a-49d2-b5cd-9ecb56d7f331\") "
Nov 03 01:15:34.373710 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.373321    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:34.377520 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.377486    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:34.377761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.377720    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:34.377944 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.377810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:34.378104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.378084    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:34.378760 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.378734    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:34.380343 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.380311    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:34.380647 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.380622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "kube-api-access-4w669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:34.383275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.383105    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:15:34.384452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.384407    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:34.384452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.384431    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2764d1af-7e0f-40d0-82e3-5e4c37ad79c5" containerName="cleanup"
Nov 03 01:15:34.384619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.384483    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy"
Nov 03 01:15:34.384619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.384492    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy"
Nov 03 01:15:34.384737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.384689    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerName="istio-proxy"
Nov 03 01:15:34.390059 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.390033    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.391390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.391348    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo" pod "0e582b53-d78a-49d2-b5cd-9ecb56d7f331" (UID: "0e582b53-d78a-49d2-b5cd-9ecb56d7f331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:15:34.396704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.396681    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3c5983-openshift-default-dockercfg-jz9jw\""
Nov 03 01:15:34.401203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.401177    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:15:34.474502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.474464    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.474702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.474649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.474702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.474691    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.474862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.474834    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.474931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.474879    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.475039 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475023    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.475097 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475082    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.475235 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmp8\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.475348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475331    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.475494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475479    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475500    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475515    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475530    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475543    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475557    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475573    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475587    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.475677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.475601    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w669\" (UniqueName: \"kubernetes.io/projected/0e582b53-d78a-49d2-b5cd-9ecb56d7f331-kube-api-access-4w669\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:34.577136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577100    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577296    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577428 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577340    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577428 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577397    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577428    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577532    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577548    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577679 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577616    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577744 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577721    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577728    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmp8\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577797 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577758    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577899 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.577964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.577942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.578280 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.578250    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.579880 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.579859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.580152 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.580135    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.585680 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.585653    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.585805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.585757    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmp8\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\") pod \"e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") " pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.723893 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.723548    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:34.832229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.832194    4531 generic.go:356] "Generic (PLEG): container finished" podID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerID="89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc" exitCode=60
Nov 03 01:15:34.832596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.832283    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerDied","Data":"89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"}
Nov 03 01:15:34.832596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.832329    4531 scope.go:122] "RemoveContainer" containerID="e45e82c0245362cb26c30a89f570a2880755b6961e273d699c1cd4336c2c2721"
Nov 03 01:15:34.832801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.832786    4531 scope.go:122] "RemoveContainer" containerID="89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"
Nov 03 01:15:34.833390 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:34.833355    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb_odh-ai-gateway-infra(6cece9b7-6774-4168-aea5-0a86688d01cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd"
Nov 03 01:15:34.839148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.839125    4531 scope.go:122] "RemoveContainer" containerID="5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"
Nov 03 01:15:34.839518 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:34.839491    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-qrpq7_odh-ai-gateway-infra(2a45e1bd-4889-404f-9b3b-4a52cddaec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90"
Nov 03 01:15:34.841335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.841262    4531 generic.go:356] "Generic (PLEG): container finished" podID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" containerID="343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518" exitCode=0
Nov 03 01:15:34.841335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.841313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" event={"ID":"0e582b53-d78a-49d2-b5cd-9ecb56d7f331","Type":"ContainerDied","Data":"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"}
Nov 03 01:15:34.841474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.841358    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln" event={"ID":"0e582b53-d78a-49d2-b5cd-9ecb56d7f331","Type":"ContainerDied","Data":"24dccc77cc644fac11388bdcd916bde8f21115cd0c234f94d7d1bcafac72c754"}
Nov 03 01:15:34.841474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.841434    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"
Nov 03 01:15:34.906938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.905182    4531 scope.go:122] "RemoveContainer" containerID="343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"
Nov 03 01:15:34.996774 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:34.996692    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 01:15:35.003037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.002464    4531 scope.go:122] "RemoveContainer" containerID="343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"
Nov 03 01:15:35.003037 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:35.003025    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518\": container with ID starting with 343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518 not found: ID does not exist" containerID="343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"
Nov 03 01:15:35.003289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.003065    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518"} err="failed to get container status \"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518\": rpc error: code = NotFound desc = could not find container \"343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518\": container with ID starting with 343256a42723163077a1c388569418903cd49e5b4fbd37bbb9f794e16e59b518 not found: ID does not exist"
Nov 03 01:15:35.008326 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.008295    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-c5gln"]
Nov 03 01:15:35.058302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.057082    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:15:35.087073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.086752    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:35.087073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.086877    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:35.087073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.086949    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:35.508254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.505751    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:35.536694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.536652    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:35.537506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537485    4531 cpu_manager.go:414] "containerMap: removing container" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537503    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537514    4531 cpu_manager.go:414] "containerMap: removing container" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537520    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537660    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537676    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.537707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.537683    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:15:35.543195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.541385    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.554733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.554699    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:35.614846 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.614807    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\") pod \"17cc790e-ec67-48fb-b4e5-583caf820577\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") "
Nov 03 01:15:35.615026 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.614880    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\" (UniqueName: \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\") pod \"17cc790e-ec67-48fb-b4e5-583caf820577\" (UID: \"17cc790e-ec67-48fb-b4e5-583caf820577\") "
Nov 03 01:15:35.615273 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.615243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca" pod "17cc790e-ec67-48fb-b4e5-583caf820577" (UID: "17cc790e-ec67-48fb-b4e5-583caf820577"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:35.615454 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.615430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5v8\" (UniqueName: \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.615606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.615592    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.615856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.615839    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17cc790e-ec67-48fb-b4e5-583caf820577-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:35.617654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.617628    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd" pod "17cc790e-ec67-48fb-b4e5-583caf820577" (UID: "17cc790e-ec67-48fb-b4e5-583caf820577"). InnerVolumeSpecName "kube-api-access-7f4qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:35.717527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.717486    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5v8\" (UniqueName: \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.717752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.717727    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.717916 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.717898    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4qd\" (UniqueName: \"kubernetes.io/projected/17cc790e-ec67-48fb-b4e5-583caf820577-kube-api-access-7f4qd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:35.718414 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.718397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.725284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.725257    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5v8\" (UniqueName: \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.855576 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.855534    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" event={"ID":"3aba7506-9c5f-4bef-baf1-66b4a98065bc","Type":"ContainerStarted","Data":"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"}
Nov 03 01:15:35.856037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.855605    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" event={"ID":"3aba7506-9c5f-4bef-baf1-66b4a98065bc","Type":"ContainerStarted","Data":"5d60d838212e79ad630568f4484abbf1ec6dbeb9a1607607b9c5870114447a10"}
Nov 03 01:15:35.857244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.857217    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"
Nov 03 01:15:35.857380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.857282    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q" event={"ID":"17cc790e-ec67-48fb-b4e5-583caf820577","Type":"ContainerDied","Data":"14851f8810c041dfb995266463b79cd0e12b15b135228de484c2d65972f24230"}
Nov 03 01:15:35.857380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.857351    4531 scope.go:122] "RemoveContainer" containerID="22c8179bc72bd972898287a4fc17671e2ae59528fdbbbd3808c94bb42ea7175f"
Nov 03 01:15:35.860542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.860519    4531 scope.go:122] "RemoveContainer" containerID="89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"
Nov 03 01:15:35.860924 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:35.860866    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb_odh-ai-gateway-infra(6cece9b7-6774-4168-aea5-0a86688d01cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd"
Nov 03 01:15:35.876458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.876371    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:35.880676 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.880623    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podStartSLOduration=6.880608334 podStartE2EDuration="6.880608334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:35.876656208 +0000 UTC m=+2081.090399881" watchObservedRunningTime="2026-11-03 01:15:35.880608334 +0000 UTC m=+2081.094352006"
Nov 03 01:15:35.911859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.911823    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:35.917556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:35.917526    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894475-jzm4q"]
Nov 03 01:15:36.175709 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:15:36.175671    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea33a055_6ea1_4c7a_a4f6_534e48502018.slice/crio-995e7e006a53adc99fa795456439eb136028cd4162ca9c19ecf69d05b06bba19 WatchSource:0}: Error finding container 995e7e006a53adc99fa795456439eb136028cd4162ca9c19ecf69d05b06bba19: Status 404 returned error can't find the container with id 995e7e006a53adc99fa795456439eb136028cd4162ca9c19ecf69d05b06bba19
Nov 03 01:15:36.175868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.175755    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:36.724393 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.724355    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:36.754182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.743271    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:36.754182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.743378    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:36.879948 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.879913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerStarted","Data":"4f4e453f4074506188344001910257d8b5350f1ad1fe8b53af956f30d9d59ee3"}
Nov 03 01:15:36.880494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.880464    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerStarted","Data":"995e7e006a53adc99fa795456439eb136028cd4162ca9c19ecf69d05b06bba19"}
Nov 03 01:15:36.897493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.897442    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" podStartSLOduration=36.897427591 podStartE2EDuration="36.897427591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:36.895796357 +0000 UTC m=+2082.109540025" watchObservedRunningTime="2026-11-03 01:15:36.897427591 +0000 UTC m=+2082.111171263"
Nov 03 01:15:36.906291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.906261    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e582b53-d78a-49d2-b5cd-9ecb56d7f331" path="/var/lib/kubelet/pods/0e582b53-d78a-49d2-b5cd-9ecb56d7f331/volumes"
Nov 03 01:15:36.906793 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:36.906777    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" path="/var/lib/kubelet/pods/17cc790e-ec67-48fb-b4e5-583caf820577/volumes"
Nov 03 01:15:37.724660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:37.724573    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:37.724660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:37.724626    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:37.894393 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:37.894356    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerID="4f4e453f4074506188344001910257d8b5350f1ad1fe8b53af956f30d9d59ee3" exitCode=60
Nov 03 01:15:37.894828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:37.894433    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerDied","Data":"4f4e453f4074506188344001910257d8b5350f1ad1fe8b53af956f30d9d59ee3"}
Nov 03 01:15:37.894896 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:37.894879    4531 scope.go:122] "RemoveContainer" containerID="4f4e453f4074506188344001910257d8b5350f1ad1fe8b53af956f30d9d59ee3"
Nov 03 01:15:38.724583 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:38.724529    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:38.724772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:38.724618    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:38.905936 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:38.905895    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerStarted","Data":"af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"}
Nov 03 01:15:39.724817 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:39.724780    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:39.725003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:39.724834    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:39.940930 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:39.940891    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:15:40.282675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.279293    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerID="af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0" exitCode=60
Nov 03 01:15:40.282675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.279360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerDied","Data":"af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"}
Nov 03 01:15:40.282675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.279418    4531 scope.go:122] "RemoveContainer" containerID="4f4e453f4074506188344001910257d8b5350f1ad1fe8b53af956f30d9d59ee3"
Nov 03 01:15:40.282675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.279989    4531 scope.go:122] "RemoveContainer" containerID="af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"
Nov 03 01:15:40.282675 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:40.280484    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm_odh-ai-gateway-infra(ea33a055-6ea1-4c7a-a4f6-534e48502018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018"
Nov 03 01:15:40.724649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.724612    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:40.724950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.724913    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:40.897380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:40.897353    4531 scope.go:122] "RemoveContainer" containerID="e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"
Nov 03 01:15:41.290534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:41.290504    4531 scope.go:122] "RemoveContainer" containerID="af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"
Nov 03 01:15:41.290988 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:41.290908    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm_odh-ai-gateway-infra(ea33a055-6ea1-4c7a-a4f6-534e48502018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018"
Nov 03 01:15:41.724796 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:41.724694    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 01:15:41.724796 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:41.724770    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 01:15:42.301644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.301613    4531 generic.go:356] "Generic (PLEG): container finished" podID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerID="e1207c2c952cf0bd8b66ebaba99d5f1fe4e01264443b5b9366fe1be242089d95" exitCode=60
Nov 03 01:15:42.302150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.301688    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerDied","Data":"e1207c2c952cf0bd8b66ebaba99d5f1fe4e01264443b5b9366fe1be242089d95"}
Nov 03 01:15:42.302150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.301764    4531 scope.go:122] "RemoveContainer" containerID="e2a2bcba194125732a59b49d0e9dc790c20d8f6371728a2acd6c45640435d4ad"
Nov 03 01:15:42.302346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.302272    4531 scope.go:122] "RemoveContainer" containerID="e1207c2c952cf0bd8b66ebaba99d5f1fe4e01264443b5b9366fe1be242089d95"
Nov 03 01:15:42.302746 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:42.302705    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp_odh-ai-gateway-infra(353e8d9e-eb46-480c-b3d2-715ab122093c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c"
Nov 03 01:15:42.730802 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.730766    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:42.731803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.731777    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:42.732622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:42.732591    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:15:43.342024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.341983    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:43.879678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.879643    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:43.986351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.986257    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\" (UniqueName: \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\") pod \"353e8d9e-eb46-480c-b3d2-715ab122093c\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") "
Nov 03 01:15:43.986534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.986434    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\") pod \"353e8d9e-eb46-480c-b3d2-715ab122093c\" (UID: \"353e8d9e-eb46-480c-b3d2-715ab122093c\") "
Nov 03 01:15:43.987051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.987018    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca" pod "353e8d9e-eb46-480c-b3d2-715ab122093c" (UID: "353e8d9e-eb46-480c-b3d2-715ab122093c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:43.987335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.987314    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/353e8d9e-eb46-480c-b3d2-715ab122093c-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:43.989300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:43.989271    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z" pod "353e8d9e-eb46-480c-b3d2-715ab122093c" (UID: "353e8d9e-eb46-480c-b3d2-715ab122093c"). InnerVolumeSpecName "kube-api-access-6mw4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:44.089486 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.089442    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mw4z\" (UniqueName: \"kubernetes.io/projected/353e8d9e-eb46-480c-b3d2-715ab122093c-kube-api-access-6mw4z\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:44.453206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.444046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"
Nov 03 01:15:44.481147 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.481108    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp" event={"ID":"353e8d9e-eb46-480c-b3d2-715ab122093c","Type":"ContainerDied","Data":"0f12bf4cc749449dce22e4b8d784beaabff4b0054f6034b8012d2ef42dd8c65e"}
Nov 03 01:15:44.481429 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.481410    4531 scope.go:122] "RemoveContainer" containerID="e1207c2c952cf0bd8b66ebaba99d5f1fe4e01264443b5b9366fe1be242089d95"
Nov 03 01:15:44.603682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.603645    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:44.631182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.629667    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d25759-29894475-rj2rp"]
Nov 03 01:15:44.909825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:44.909781    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" path="/var/lib/kubelet/pods/353e8d9e-eb46-480c-b3d2-715ab122093c/volumes"
Nov 03 01:15:46.896741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:46.896702    4531 scope.go:122] "RemoveContainer" containerID="5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"
Nov 03 01:15:47.470873 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:47.470841    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerStarted","Data":"44d683eb3f1fe8d79fda5cdb32ed070d9732def4c3dfd9692f45db8164b69d93"}
Nov 03 01:15:47.896962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:47.896930    4531 scope.go:122] "RemoveContainer" containerID="89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"
Nov 03 01:15:48.481823 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.481785    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerStarted","Data":"1e9a216359235839ab7fb8d50b77ae6567997831c7ce0718659641702d1d35f8"}
Nov 03 01:15:48.483943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.483913    4531 generic.go:356] "Generic (PLEG): container finished" podID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerID="44d683eb3f1fe8d79fda5cdb32ed070d9732def4c3dfd9692f45db8164b69d93" exitCode=60
Nov 03 01:15:48.484076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.483986    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerDied","Data":"44d683eb3f1fe8d79fda5cdb32ed070d9732def4c3dfd9692f45db8164b69d93"}
Nov 03 01:15:48.484076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.484041    4531 scope.go:122] "RemoveContainer" containerID="5a806620df89f9ae57680e22940828fc20c05263895d81617fbf6e11d2a46b5a"
Nov 03 01:15:48.484483 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.484463    4531 scope.go:122] "RemoveContainer" containerID="44d683eb3f1fe8d79fda5cdb32ed070d9732def4c3dfd9692f45db8164b69d93"
Nov 03 01:15:48.484894 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:48.484870    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-qrpq7_odh-ai-gateway-infra(2a45e1bd-4889-404f-9b3b-4a52cddaec90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90"
Nov 03 01:15:48.512573 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:48.512545    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:49.497648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.497613    4531 generic.go:356] "Generic (PLEG): container finished" podID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerID="1e9a216359235839ab7fb8d50b77ae6567997831c7ce0718659641702d1d35f8" exitCode=60
Nov 03 01:15:49.498252 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.497687    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerDied","Data":"1e9a216359235839ab7fb8d50b77ae6567997831c7ce0718659641702d1d35f8"}
Nov 03 01:15:49.498252 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.497749    4531 scope.go:122] "RemoveContainer" containerID="89dfaaf1fbb81e0c2aa3eb8e0df187191e595733cbf2e9ada73024c8d06e69fc"
Nov 03 01:15:49.498390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.498299    4531 scope.go:122] "RemoveContainer" containerID="1e9a216359235839ab7fb8d50b77ae6567997831c7ce0718659641702d1d35f8"
Nov 03 01:15:49.499090 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:49.498935    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb_odh-ai-gateway-infra(6cece9b7-6774-4168-aea5-0a86688d01cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd"
Nov 03 01:15:49.531575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.531543    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:49.728088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.728049    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:49.821915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.821885    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\" (UniqueName: \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\") pod \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") "
Nov 03 01:15:49.822309 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.822290    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\") pod \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\" (UID: \"2a45e1bd-4889-404f-9b3b-4a52cddaec90\") "
Nov 03 01:15:49.822689 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.822665    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca" pod "2a45e1bd-4889-404f-9b3b-4a52cddaec90" (UID: "2a45e1bd-4889-404f-9b3b-4a52cddaec90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:49.822956 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.822938    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a45e1bd-4889-404f-9b3b-4a52cddaec90-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:49.824705 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.824684    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl" pod "2a45e1bd-4889-404f-9b3b-4a52cddaec90" (UID: "2a45e1bd-4889-404f-9b3b-4a52cddaec90"). InnerVolumeSpecName "kube-api-access-mp4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:49.924897 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:49.924859    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4dl\" (UniqueName: \"kubernetes.io/projected/2a45e1bd-4889-404f-9b3b-4a52cddaec90-kube-api-access-mp4dl\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:50.513622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.513596    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"
Nov 03 01:15:50.514021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.513595    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7" event={"ID":"2a45e1bd-4889-404f-9b3b-4a52cddaec90","Type":"ContainerDied","Data":"a201410afff2c7baf18e9f178e5ac1fe5e7c7f59959131c83f57b24736ad2e58"}
Nov 03 01:15:50.514021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.513739    4531 scope.go:122] "RemoveContainer" containerID="44d683eb3f1fe8d79fda5cdb32ed070d9732def4c3dfd9692f45db8164b69d93"
Nov 03 01:15:50.552710 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.552662    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:50.562678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.562646    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-qrpq7"]
Nov 03 01:15:50.721828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.721800    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:50.849761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.849731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\" (UniqueName: \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\") pod \"6cece9b7-6774-4168-aea5-0a86688d01cd\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") "
Nov 03 01:15:50.849913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.849815    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\") pod \"6cece9b7-6774-4168-aea5-0a86688d01cd\" (UID: \"6cece9b7-6774-4168-aea5-0a86688d01cd\") "
Nov 03 01:15:50.850277 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.850245    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca" pod "6cece9b7-6774-4168-aea5-0a86688d01cd" (UID: "6cece9b7-6774-4168-aea5-0a86688d01cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:50.850467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.850411    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cece9b7-6774-4168-aea5-0a86688d01cd-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:50.851922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.851901    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2" pod "6cece9b7-6774-4168-aea5-0a86688d01cd" (UID: "6cece9b7-6774-4168-aea5-0a86688d01cd"). InnerVolumeSpecName "kube-api-access-cg7q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:50.896487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.896453    4531 scope.go:122] "RemoveContainer" containerID="af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"
Nov 03 01:15:50.912904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.912873    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" path="/var/lib/kubelet/pods/2a45e1bd-4889-404f-9b3b-4a52cddaec90/volumes"
Nov 03 01:15:50.952451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:50.952419    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7q2\" (UniqueName: \"kubernetes.io/projected/6cece9b7-6774-4168-aea5-0a86688d01cd-kube-api-access-cg7q2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:51.547240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.546517    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerStarted","Data":"b235841bcb280e44ae8610ff8fbd0c5c495d4019a170f716b996e0541ca249ce"}
Nov 03 01:15:51.574548 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.557179    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb" event={"ID":"6cece9b7-6774-4168-aea5-0a86688d01cd","Type":"ContainerDied","Data":"bd96cc058accd7a6934b0bf0b9d32301423c0e61df22dacf7f733afc59ef4f7e"}
Nov 03 01:15:51.574548 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.557256    4531 scope.go:122] "RemoveContainer" containerID="1e9a216359235839ab7fb8d50b77ae6567997831c7ce0718659641702d1d35f8"
Nov 03 01:15:51.574548 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.557352    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"
Nov 03 01:15:51.702394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.698097    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:51.749335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:51.749117    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ddd653-29894475-s56tb"]
Nov 03 01:15:52.600112 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.600077    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerID="b235841bcb280e44ae8610ff8fbd0c5c495d4019a170f716b996e0541ca249ce" exitCode=60
Nov 03 01:15:52.600600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.600152    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerDied","Data":"b235841bcb280e44ae8610ff8fbd0c5c495d4019a170f716b996e0541ca249ce"}
Nov 03 01:15:52.600600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.600251    4531 scope.go:122] "RemoveContainer" containerID="af65ee8960f0ce72e0afb8e4b9b19d6bae3ab241084c02607edaa350fbcb97c0"
Nov 03 01:15:52.600835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.600820    4531 scope.go:122] "RemoveContainer" containerID="b235841bcb280e44ae8610ff8fbd0c5c495d4019a170f716b996e0541ca249ce"
Nov 03 01:15:52.601484 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:15:52.601447    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm_odh-ai-gateway-infra(ea33a055-6ea1-4c7a-a4f6-534e48502018)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018"
Nov 03 01:15:52.610776 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.610750    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:52.904937 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:52.904859    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" path="/var/lib/kubelet/pods/6cece9b7-6774-4168-aea5-0a86688d01cd/volumes"
Nov 03 01:15:53.803299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.803273    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:53.960963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.960879    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\" (UniqueName: \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\") pod \"ea33a055-6ea1-4c7a-a4f6-534e48502018\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") "
Nov 03 01:15:53.961151 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.961134    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\") pod \"ea33a055-6ea1-4c7a-a4f6-534e48502018\" (UID: \"ea33a055-6ea1-4c7a-a4f6-534e48502018\") "
Nov 03 01:15:53.961579 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.961541    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca" pod "ea33a055-6ea1-4c7a-a4f6-534e48502018" (UID: "ea33a055-6ea1-4c7a-a4f6-534e48502018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:53.961957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.961935    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea33a055-6ea1-4c7a-a4f6-534e48502018-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:53.963249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:53.963219    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8" pod "ea33a055-6ea1-4c7a-a4f6-534e48502018" (UID: "ea33a055-6ea1-4c7a-a4f6-534e48502018"). InnerVolumeSpecName "kube-api-access-xs5v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:54.082349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.064708    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5v8\" (UniqueName: \"kubernetes.io/projected/ea33a055-6ea1-4c7a-a4f6-534e48502018-kube-api-access-xs5v8\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:15:54.624768 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.624737    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"
Nov 03 01:15:54.624974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.624737    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm" event={"ID":"ea33a055-6ea1-4c7a-a4f6-534e48502018","Type":"ContainerDied","Data":"995e7e006a53adc99fa795456439eb136028cd4162ca9c19ecf69d05b06bba19"}
Nov 03 01:15:54.624974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.624884    4531 scope.go:122] "RemoveContainer" containerID="b235841bcb280e44ae8610ff8fbd0c5c495d4019a170f716b996e0541ca249ce"
Nov 03 01:15:54.661336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.661303    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:54.665761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.665729    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894475-2fjdm"]
Nov 03 01:15:54.906058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:15:54.905981    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" path="/var/lib/kubelet/pods/ea33a055-6ea1-4c7a-a4f6-534e48502018/volumes"
Nov 03 01:16:01.657187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:01.637947    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 01:16:01.657187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:01.638271    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" containerID="cri-o://d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281" gracePeriod=30
Nov 03 01:16:02.786869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:02.786827    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 01:16:02.787386 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:02.787048    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy" containerID="cri-o://7655805a17089be38bddf0cd55a5baed1b2c125c8d2f3ecc72fdb93e3d07ab69" gracePeriod=30
Nov 03 01:16:03.850670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:03.850620    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-skx44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:16:03.851111 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:03.850710    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:16:07.021384 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.021352    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 01:16:07.048364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.047293    4531 generic.go:356] "Generic (PLEG): container finished" podID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerID="d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281" exitCode=0
Nov 03 01:16:07.048364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.047417    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"
Nov 03 01:16:07.048364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.047786    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerDied","Data":"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"}
Nov 03 01:16:07.048669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.048635    4531 scope.go:122] "RemoveContainer" containerID="d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"
Nov 03 01:16:07.051326 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.051258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44" event={"ID":"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44","Type":"ContainerDied","Data":"b6b9f056d114fb8f9029434b35a67323ee57dd8e2f201a06267ee00b2f62ffd9"}
Nov 03 01:16:07.057907 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.057876    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:16:07.058988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.058959    4531 cpu_manager.go:414] "containerMap: removing container" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:16:07.059077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.058990    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17cc790e-ec67-48fb-b4e5-583caf820577" containerName="cleanup"
Nov 03 01:16:07.059077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059017    4531 cpu_manager.go:414] "containerMap: removing container" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.059077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059025    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.059077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059051    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059060    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059082    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059090    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059109    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059116    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059130    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059136    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059144    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059151    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059180    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059187    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059201    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059208    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059253    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059261    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059270    4531 cpu_manager.go:414] "containerMap: removing container" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059278    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059288    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059297    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059311    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.059379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059318    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059509    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059526    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059535    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059545    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059556    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059568    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059578    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" containerName="istio-proxy"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059589    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059598    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059605    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a45e1bd-4889-404f-9b3b-4a52cddaec90" containerName="cleanup"
Nov 03 01:16:07.060418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.059618    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:07.070438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.067390    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.070438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.069014    4531 scope.go:122] "RemoveContainer" containerID="761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"
Nov 03 01:16:07.078580 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.078497    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:16:07.088711 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.086513    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-af8244-openshift-default-dockercfg-7hb8r\""
Nov 03 01:16:07.090830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.090806    4531 scope.go:122] "RemoveContainer" containerID="d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"
Nov 03 01:16:07.091262 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:16:07.091232    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281\": container with ID starting with d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281 not found: ID does not exist" containerID="d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"
Nov 03 01:16:07.091364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.091269    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281"} err="failed to get container status \"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281\": rpc error: code = NotFound desc = could not find container \"d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281\": container with ID starting with d749850c7663250c0e2645ca7c93f53306bafa3307d0d98eeb0fac2fe4c30281 not found: ID does not exist"
Nov 03 01:16:07.091364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.091297    4531 scope.go:122] "RemoveContainer" containerID="761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"
Nov 03 01:16:07.091996 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:16:07.091966    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed\": container with ID starting with 761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed not found: ID does not exist" containerID="761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"
Nov 03 01:16:07.092067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.092008    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed"} err="failed to get container status \"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed\": rpc error: code = NotFound desc = could not find container \"761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed\": container with ID starting with 761acdf34a1e498aa6d1c7ff84362430d9a7739e0d3a7f44f8fd8aee5581ceed not found: ID does not exist"
Nov 03 01:16:07.148021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.147989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148112    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148152    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148212    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148286    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148400    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148518 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148633 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148614    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148619    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:07.148708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\") pod \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\" (UID: \"4f5c6128-d29a-4fd2-a4b0-c01e2d068c44\") "
Nov 03 01:16:07.148879 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148834    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:07.149002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.148982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:07.149212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149176    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:07.149291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149211    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149485    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149573    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149640    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149669    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7mx\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149809    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.149982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.149896    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.150134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150026    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.150134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150044    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.150134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150062    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.150134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150076    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.150134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:07.151222 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.150988    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:07.151222 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.151134    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:07.155329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.155295    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "kube-api-access-ll5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:07.155791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.155765    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy" pod "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" (UID: "4f5c6128-d29a-4fd2-a4b0-c01e2d068c44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:07.251296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251139    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251263    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251284    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251396    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251567    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251681    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7mx\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251770    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251786    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251789    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251844    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll5hp\" (UniqueName: \"kubernetes.io/projected/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-kube-api-access-ll5hp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251848    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251861    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.251913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.251901    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:07.252352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.252139    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.252352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.252248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.252534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.252512    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.254175 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.254138    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.254398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.254378    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.259123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.259098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7mx\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.259248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.259122    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\") pod \"e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") " pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.382284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.382246    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 01:16:07.388499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.388467    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-skx44"]
Nov 03 01:16:07.398878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.398848    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:07.757095 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:16:07.757048    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda667d4b4_e891_4402_98d5_55a6edb85f51.slice/crio-efb36d14926ab94ee5f3bf08a806326b35024a486b6e9a8dc733ea8806c0519c WatchSource:0}: Error finding container efb36d14926ab94ee5f3bf08a806326b35024a486b6e9a8dc733ea8806c0519c: Status 404 returned error can't find the container with id efb36d14926ab94ee5f3bf08a806326b35024a486b6e9a8dc733ea8806c0519c
Nov 03 01:16:07.757432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.757397    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:16:07.760855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.760813    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:07.760989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.760908    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:07.760989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:07.760951    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:08.064248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.063592    4531 generic.go:356] "Generic (PLEG): container finished" podID="4163426a-911a-44e4-b50d-f30c2fe24131" containerID="7655805a17089be38bddf0cd55a5baed1b2c125c8d2f3ecc72fdb93e3d07ab69" exitCode=0
Nov 03 01:16:08.064248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.063797    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" event={"ID":"4163426a-911a-44e4-b50d-f30c2fe24131","Type":"ContainerDied","Data":"7655805a17089be38bddf0cd55a5baed1b2c125c8d2f3ecc72fdb93e3d07ab69"}
Nov 03 01:16:08.068318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.067220    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" event={"ID":"a667d4b4-e891-4402-98d5-55a6edb85f51","Type":"ContainerStarted","Data":"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"}
Nov 03 01:16:08.068318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.067283    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" event={"ID":"a667d4b4-e891-4402-98d5-55a6edb85f51","Type":"ContainerStarted","Data":"efb36d14926ab94ee5f3bf08a806326b35024a486b6e9a8dc733ea8806c0519c"}
Nov 03 01:16:08.116192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.102740    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podStartSLOduration=40.102715061 podStartE2EDuration="40.102715061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:08.093588427 +0000 UTC m=+2113.307332100" watchObservedRunningTime="2026-11-03 01:16:08.102715061 +0000 UTC m=+2113.316458735"
Nov 03 01:16:08.215954 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.215926    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 01:16:08.252780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.252731    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:16:08.254111 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254010    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:08.254312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254285    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:08.254434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254327    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy"
Nov 03 01:16:08.254434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254336    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy"
Nov 03 01:16:08.254434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254353    4531 cpu_manager.go:414] "containerMap: removing container" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:08.254434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254360    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:08.254678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254609    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cece9b7-6774-4168-aea5-0a86688d01cd" containerName="cleanup"
Nov 03 01:16:08.254678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254622    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" containerName="istio-proxy"
Nov 03 01:16:08.254678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254635    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="353e8d9e-eb46-480c-b3d2-715ab122093c" containerName="cleanup"
Nov 03 01:16:08.254678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.254652    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea33a055-6ea1-4c7a-a4f6-534e48502018" containerName="cleanup"
Nov 03 01:16:08.259140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.259111    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.268017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.263650    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e1f08d-openshift-default-dockercfg-kd24b\""
Nov 03 01:16:08.268017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.267366    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:16:08.303718 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303672    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.303896 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303758    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.303896 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303795    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303956    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.303994    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.304051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304013    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304285    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304368    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\") pod \"4163426a-911a-44e4-b50d-f30c2fe24131\" (UID: \"4163426a-911a-44e4-b50d-f30c2fe24131\") "
Nov 03 01:16:08.304733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304704    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.304840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304777    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.304945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.304945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304928    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304960    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.305065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.304997    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305032    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrn8\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305267 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305184    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305279    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305294    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:08.305618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305595    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305639    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.305774 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.306014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305908    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4163426a-911a-44e4-b50d-f30c2fe24131-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.306014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305929    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.306014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305944    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.306014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305958    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.306014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.305972    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.307389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.307357    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.307487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.307430    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:08.307727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.307702    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:08.308100 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.308069    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr" pod "4163426a-911a-44e4-b50d-f30c2fe24131" (UID: "4163426a-911a-44e4-b50d-f30c2fe24131"). InnerVolumeSpecName "kube-api-access-xbhcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:08.399867 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.399835    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:08.401415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.401384    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:08.401538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.401438    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:08.407041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407013    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407176    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407198    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407313 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407374 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407392    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407471    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrn8\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407549    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407793    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.407993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407968    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.407993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.407997    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4163426a-911a-44e4-b50d-f30c2fe24131-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.408229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.408013    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4163426a-911a-44e4-b50d-f30c2fe24131-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.408229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.408029    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhcr\" (UniqueName: \"kubernetes.io/projected/4163426a-911a-44e4-b50d-f30c2fe24131-kube-api-access-xbhcr\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:16:08.408229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.408184    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.410507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.410488    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.410835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.410813    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.419753 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.419720    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.420002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.419974    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrn8\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\") pod \"e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") " pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.588186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.588125    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:08.918452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.917847    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:08.918452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.917932    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:08.918452 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.917976    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:16:08.925742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.925659    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5c6128-d29a-4fd2-a4b0-c01e2d068c44" path="/var/lib/kubelet/pods/4f5c6128-d29a-4fd2-a4b0-c01e2d068c44/volumes"
Nov 03 01:16:08.952240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:08.950364    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:16:09.083183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.081645    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" event={"ID":"37be08c0-1f71-43a8-869e-bc918ba4088c","Type":"ContainerStarted","Data":"dcd828d33d491d7c5fa06e60363d4abb3ffd074e816bc1b43825256668f197f7"}
Nov 03 01:16:09.083661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.083332    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"
Nov 03 01:16:09.083661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.083341    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r" event={"ID":"4163426a-911a-44e4-b50d-f30c2fe24131","Type":"ContainerDied","Data":"c82b73d1a750aa2a9cdaee1722d53d8895dc970ac3286074eedf2690dac3229e"}
Nov 03 01:16:09.083661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.083396    4531 scope.go:122] "RemoveContainer" containerID="7655805a17089be38bddf0cd55a5baed1b2c125c8d2f3ecc72fdb93e3d07ab69"
Nov 03 01:16:09.108732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.108671    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 01:16:09.114704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.114673    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-whq6r"]
Nov 03 01:16:09.402803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.400391    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:09.402803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:09.400451    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:10.105428 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:16:10.099479    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:16:10.217339 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.217303    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" event={"ID":"37be08c0-1f71-43a8-869e-bc918ba4088c","Type":"ContainerStarted","Data":"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"}
Nov 03 01:16:10.319033 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.307976    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podStartSLOduration=59.307955452 podStartE2EDuration="59.307955452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:10.301043796 +0000 UTC m=+2115.514787469" watchObservedRunningTime="2026-11-03 01:16:10.307955452 +0000 UTC m=+2115.521699126"
Nov 03 01:16:10.415241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.414571    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:10.415241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.414662    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:10.600269 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.600227    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:10.643349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.643304    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:10.643621 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:10.643579    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:11.028257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.028218    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4163426a-911a-44e4-b50d-f30c2fe24131" path="/var/lib/kubelet/pods/4163426a-911a-44e4-b50d-f30c2fe24131/volumes"
Nov 03 01:16:11.405015 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.404974    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:11.407290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.405610    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:11.512232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.510267    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:16:11.596193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.588624    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:11.596193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:11.588689    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:12.400467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:12.400419    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:12.400654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:12.400484    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:12.589671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:12.589629    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:12.590292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:12.590249    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:13.399868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:13.399830    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:13.400270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:13.399887    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:13.591761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:13.591717    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:13.592327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:13.591796    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:14.408192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:14.401584    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 01:16:14.408192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:14.401651    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 01:16:14.589595 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:14.589552    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:14.589886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:14.589849    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:15.593302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:15.593262    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:16:15.593910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:15.593876    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:16:15.696964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:15.696909    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:15.699479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:15.699019    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:15.762565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:15.762504    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:16:16.593951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:16.593912    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:16.594463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:16.594439    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:16.595260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:16.595243    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:16:20.967758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:20.967718    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:16:20.972423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:20.972397    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:20.978444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:20.978413    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-be59df\""
Nov 03 01:16:20.978598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:20.978586    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-be59df-dockercfg-l2wrh\""
Nov 03 01:16:20.986754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:20.985822    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:16:21.133184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.130568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqc4p\" (UniqueName: \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.133184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.130907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.261236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.257428    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqc4p\" (UniqueName: \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.261236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.257606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.261236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.258682    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.301360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.301325    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqc4p\" (UniqueName: \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\") pod \"payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.314274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.310490    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:21.775739 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.775705    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:16:21.776222 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:16:21.776191    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b082ef0_f9b7_4096_a60c_0da0636c1846.slice/crio-113c0c560c461322b6ab83de2dd180c961355fb85a02d1ff075dbabccc54302e WatchSource:0}: Error finding container 113c0c560c461322b6ab83de2dd180c961355fb85a02d1ff075dbabccc54302e: Status 404 returned error can't find the container with id 113c0c560c461322b6ab83de2dd180c961355fb85a02d1ff075dbabccc54302e
Nov 03 01:16:21.778436 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:21.778409    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:16:22.523695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:22.523659    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" event={"ID":"1b082ef0-f9b7-4096-a60c-0da0636c1846","Type":"ContainerStarted","Data":"113c0c560c461322b6ab83de2dd180c961355fb85a02d1ff075dbabccc54302e"}
Nov 03 01:16:25.575196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:25.572576    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" event={"ID":"1b082ef0-f9b7-4096-a60c-0da0636c1846","Type":"ContainerStarted","Data":"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"}
Nov 03 01:16:25.575196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:25.573745    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:25.653262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:25.648485    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" podStartSLOduration=2.092990579 podStartE2EDuration="5.648461147s" totalImagesPullingTime="3.555470568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:25.619012669 +0000 UTC m=+2130.832756344" watchObservedRunningTime="2026-11-03 01:16:25.648461147 +0000 UTC m=+2130.862204819"
Nov 03 01:16:37.601799 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:16:37.601766    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:16:39.891071 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:16:39.891028    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:17:09.920456 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:17:09.920420    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:17:17.659319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:17.659273    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 01:17:17.659799 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:17.659485    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy" containerID="cri-o://222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81" gracePeriod=30
Nov 03 01:17:19.903655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:19.903621    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:17:19.904190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:19.903827    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy" containerID="cri-o://10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add" gracePeriod=30
Nov 03 01:17:22.961841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:22.961805    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 01:17:23.029603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.029603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029607    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.029858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029812    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.029923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029863    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.029981 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.029981 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.029966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.030092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030053    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:23.030092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030084    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.030213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030096    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:23.030213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030200    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.030369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030232    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\") pod \"535112b4-592c-4e5d-93a0-3628c871f421\" (UID: \"535112b4-592c-4e5d-93a0-3628c871f421\") "
Nov 03 01:17:23.030369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030286    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:23.030493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030473    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:23.030816 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.030790    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:23.031140 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.031114    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.031254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.031149    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.031254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.031222    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.031254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.031240    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.031416 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.031257    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535112b4-592c-4e5d-93a0-3628c871f421-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.032494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.032465    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:23.033176 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.033140    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "kube-api-access-jlcr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:23.033648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.033626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:23.033737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.033715    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token" pod "535112b4-592c-4e5d-93a0-3628c871f421" (UID: "535112b4-592c-4e5d-93a0-3628c871f421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:23.132972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.132930    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535112b4-592c-4e5d-93a0-3628c871f421-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.132972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.132965    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlcr4\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-kube-api-access-jlcr4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.132972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.132977    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535112b4-592c-4e5d-93a0-3628c871f421-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.133263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.132985    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535112b4-592c-4e5d-93a0-3628c871f421-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:23.210482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.210395    4531 generic.go:356] "Generic (PLEG): container finished" podID="535112b4-592c-4e5d-93a0-3628c871f421" containerID="222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81" exitCode=0
Nov 03 01:17:23.210482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.210455    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"
Nov 03 01:17:23.210728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.210471    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" event={"ID":"535112b4-592c-4e5d-93a0-3628c871f421","Type":"ContainerDied","Data":"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"}
Nov 03 01:17:23.210728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.210532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t" event={"ID":"535112b4-592c-4e5d-93a0-3628c871f421","Type":"ContainerDied","Data":"52e72a3acafb801542812bf594444c660c9f767e19ea5a61123e5ff4b725e61b"}
Nov 03 01:17:23.210728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.210556    4531 scope.go:122] "RemoveContainer" containerID="222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"
Nov 03 01:17:23.222656 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.222408    4531 scope.go:122] "RemoveContainer" containerID="222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"
Nov 03 01:17:23.222786 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:17:23.222765    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81\": container with ID starting with 222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81 not found: ID does not exist" containerID="222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"
Nov 03 01:17:23.222832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.222797    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81"} err="failed to get container status \"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81\": rpc error: code = NotFound desc = could not find container \"222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81\": container with ID starting with 222fc878f7fc9d7eec0bd4e9764990432f1c870e3c7ef5375c2ffa2260cc4e81 not found: ID does not exist"
Nov 03 01:17:23.287740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.287491    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 01:17:23.294439 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:23.294404    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-5mm5t"]
Nov 03 01:17:24.908404 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:24.908371    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535112b4-592c-4e5d-93a0-3628c871f421" path="/var/lib/kubelet/pods/535112b4-592c-4e5d-93a0-3628c871f421/volumes"
Nov 03 01:17:25.219384 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.219333    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:17:25.244064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.244023    4531 generic.go:356] "Generic (PLEG): container finished" podID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerID="10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add" exitCode=0
Nov 03 01:17:25.244316 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.244201    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"
Nov 03 01:17:25.244401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.244336    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" event={"ID":"f0d1deb8-7cc5-4b91-a115-a9f21c460393","Type":"ContainerDied","Data":"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"}
Nov 03 01:17:25.244453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.244433    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7" event={"ID":"f0d1deb8-7cc5-4b91-a115-a9f21c460393","Type":"ContainerDied","Data":"5ac6794663143b80050bbfa75ec1046fd23a3cae948b2f3fee0210c7fefe5451"}
Nov 03 01:17:25.244507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.244469    4531 scope.go:122] "RemoveContainer" containerID="10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"
Nov 03 01:17:25.261916 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.261893    4531 scope.go:122] "RemoveContainer" containerID="10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"
Nov 03 01:17:25.262461 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:17:25.262435    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add\": container with ID starting with 10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add not found: ID does not exist" containerID="10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"
Nov 03 01:17:25.262627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.262603    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add"} err="failed to get container status \"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add\": rpc error: code = NotFound desc = could not find container \"10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add\": container with ID starting with 10130b7f553d6bc79c728b0468e3e77e5b28b924f6a50acd04acca49745f8add not found: ID does not exist"
Nov 03 01:17:25.286692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.286884 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286729    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.286884 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286768    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.286884 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286922    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.286974    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:25.287237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287124    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287129    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:25.287375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287244    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287357    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287426    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\") pod \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\" (UID: \"f0d1deb8-7cc5-4b91-a115-a9f21c460393\") "
Nov 03 01:17:25.287702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287677    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:25.287878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.287855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:25.288243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.288218    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:25.288370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.288347    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.288433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.288376    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.288433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.288391    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.288433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.288404    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.289638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.289611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:25.290513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.290487    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:25.291534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.291513    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "kube-api-access-2zml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:25.293718 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.293693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token" pod "f0d1deb8-7cc5-4b91-a115-a9f21c460393" (UID: "f0d1deb8-7cc5-4b91-a115-a9f21c460393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:25.396068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.395552    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.396360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.396339    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.396474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.396459    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.397006 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.396963    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zml7\" (UniqueName: \"kubernetes.io/projected/f0d1deb8-7cc5-4b91-a115-a9f21c460393-kube-api-access-2zml7\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.397224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.397205    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d1deb8-7cc5-4b91-a115-a9f21c460393-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:25.577534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.577499    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:17:25.585308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:25.585281    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-hp4l7"]
Nov 03 01:17:26.904850 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:26.904815    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" path="/var/lib/kubelet/pods/f0d1deb8-7cc5-4b91-a115-a9f21c460393/volumes"
Nov 03 01:17:31.204810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:31.204765    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:17:31.205273 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:31.205033    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy" containerID="cri-o://92cdf306309dd596140acba9d23a13b803021a1ed221e60b3f31f349e634ad03" gracePeriod=30
Nov 03 01:17:36.345778 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.345744    4531 generic.go:356] "Generic (PLEG): container finished" podID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerID="92cdf306309dd596140acba9d23a13b803021a1ed221e60b3f31f349e634ad03" exitCode=0
Nov 03 01:17:36.346290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.345822    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" event={"ID":"762bfce8-cd1a-4dd3-ada4-64084f8d2eea","Type":"ContainerDied","Data":"92cdf306309dd596140acba9d23a13b803021a1ed221e60b3f31f349e634ad03"}
Nov 03 01:17:36.539627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.539584    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671234    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671476    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671512    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671710    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.671749    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.675325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.683948    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.684080    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\") pod \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\" (UID: \"762bfce8-cd1a-4dd3-ada4-64084f8d2eea\") "
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.684106    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "kube-api-access-xzhcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.684171    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:36.684508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.684422    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685204    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhcb\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-kube-api-access-xzhcb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685229    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685247    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685263    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685372    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:36.685923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.685591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:36.686293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.686040    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:36.688053 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.687928    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:36.689426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.689405    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo" pod "762bfce8-cd1a-4dd3-ada4-64084f8d2eea" (UID: "762bfce8-cd1a-4dd3-ada4-64084f8d2eea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:36.786980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.786887    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.786980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.786921    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.786980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.786934    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.786980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.786945    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:36.786980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:36.786953    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762bfce8-cd1a-4dd3-ada4-64084f8d2eea-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:17:37.357068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:37.357031    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"
Nov 03 01:17:37.357068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:37.357049    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2" event={"ID":"762bfce8-cd1a-4dd3-ada4-64084f8d2eea","Type":"ContainerDied","Data":"821d2ca5bec2d923f379412faf18afd5766d4fb57741716b9b1a3e30023a2781"}
Nov 03 01:17:37.357692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:37.357128    4531 scope.go:122] "RemoveContainer" containerID="92cdf306309dd596140acba9d23a13b803021a1ed221e60b3f31f349e634ad03"
Nov 03 01:17:37.379288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:37.379245    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:17:37.383871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:37.383844    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-prff2"]
Nov 03 01:17:38.951865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:17:38.951824    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" path="/var/lib/kubelet/pods/762bfce8-cd1a-4dd3-ada4-64084f8d2eea/volumes"
Nov 03 01:17:39.895642 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:17:39.895606    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:18:09.881701 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:18:09.881661    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:18:39.880025 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:18:39.879983    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:19:09.879415 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:19:09.879318    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:19:39.879750 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:19:39.879706    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:20:09.881934 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:20:09.881892    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:20:26.511184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:26.502572    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 01:20:26.520231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:26.512483    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy" containerID="cri-o://08ea36b41483f1fc91bc3a0612c8d9f5190b53ce11d9a1b435a7be5a0361f72f" gracePeriod=30
Nov 03 01:20:31.375733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.375690    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:20:31.376291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.375992    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy" containerID="cri-o://b09fdcbd8568004d0b6ce438e7f1f39229ae45f1b77f01837050ad91bbd3dae4" gracePeriod=30
Nov 03 01:20:31.994934 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.994830    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:20:31.996868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.996835    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy"
Nov 03 01:20:31.997084 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.997066    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy"
Nov 03 01:20:31.997241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.997227    4531 cpu_manager.go:414] "containerMap: removing container" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy"
Nov 03 01:20:31.997401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.997386    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy"
Nov 03 01:20:31.997588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.997576    4531 cpu_manager.go:414] "containerMap: removing container" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy"
Nov 03 01:20:31.997693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.997681    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy"
Nov 03 01:20:31.998923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.998905    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d1deb8-7cc5-4b91-a115-a9f21c460393" containerName="istio-proxy"
Nov 03 01:20:31.999050 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.999039    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="762bfce8-cd1a-4dd3-ada4-64084f8d2eea" containerName="istio-proxy"
Nov 03 01:20:31.999150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:31.999140    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="535112b4-592c-4e5d-93a0-3628c871f421" containerName="istio-proxy"
Nov 03 01:20:32.013471 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.011838    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.021769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.021739    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:20:32.033253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.031190    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"]
Nov 03 01:20:32.041284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.041260    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.053749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.053691    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"]
Nov 03 01:20:32.076738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.076701    4531 generic.go:356] "Generic (PLEG): container finished" podID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerID="08ea36b41483f1fc91bc3a0612c8d9f5190b53ce11d9a1b435a7be5a0361f72f" exitCode=0
Nov 03 01:20:32.076996 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.076977    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" event={"ID":"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2","Type":"ContainerDied","Data":"08ea36b41483f1fc91bc3a0612c8d9f5190b53ce11d9a1b435a7be5a0361f72f"}
Nov 03 01:20:32.077087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.077033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn" event={"ID":"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2","Type":"ContainerDied","Data":"49e63f212af77b38fe19a8f0db7ff6402eecc35185fda0049608a052c35e69ed"}
Nov 03 01:20:32.077087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.077057    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e63f212af77b38fe19a8f0db7ff6402eecc35185fda0049608a052c35e69ed"
Nov 03 01:20:32.086737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.086706    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.086881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.086753    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjgv\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.086881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.086810    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087007 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.086897    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d625w\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-kube-api-access-d625w\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087007 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.086934    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087125 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087072    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087316    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087401    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087481    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087508    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087566    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087710    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.087846 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087817    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087852    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b1042d-911e-4e83-b83d-dbb8096646d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.087963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.087922    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.088239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.088079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.088239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.088115    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.100508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.100477    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 01:20:32.189354 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189314    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189386    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189424    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189450    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189507    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189581    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189644    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189688    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.189975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189889    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:32.190031 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.189954    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\") pod \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\" (UID: \"1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2\") "
Nov 03 01:20:32.190125 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190072    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:32.190236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190144    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:32.190420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190394    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190448    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190506    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.190571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190518    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:20:32.190571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjgv\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.190742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190650    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d625w\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-kube-api-access-d625w\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190689    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.190912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190753    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190799    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190858    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.190912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190894    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190946    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.190976    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191014    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191246    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.191398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191261    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:32.191398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191289    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b1042d-911e-4e83-b83d-dbb8096646d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.191398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191331    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.191620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191549    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.191620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191568    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.191620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191586    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.191620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191604    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.191620    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.192201    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.192972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.193281    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194037    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:32.194132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194062    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "kube-api-access-7j9hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194236    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194342    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194474    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194505    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194551    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194731    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194740    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.194940    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.195137    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b1042d-911e-4e83-b83d-dbb8096646d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.195137    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy" pod "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" (UID: "1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:32.196321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.195415    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.196968 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.196571    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.199027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.199001    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.199239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.199216    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.201500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.201474    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjgv\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\") pod \"maas-default-gateway-openshift-default-596dc9545f-fjz5c\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.201834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.201809    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d625w\" (UniqueName: \"kubernetes.io/projected/37b1042d-911e-4e83-b83d-dbb8096646d3-kube-api-access-d625w\") pod \"maas-default-gateway-openshift-default-596dc9545f-46hnd\" (UID: \"37b1042d-911e-4e83-b83d-dbb8096646d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.293051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.292968    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.293051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.293001    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9hx\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-kube-api-access-7j9hx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.293051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.293013    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.293051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.293022    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:32.354387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.354340    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:32.399093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.399057    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:32.650233 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:20:32.650197    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d4dcef_ba0d_4f3b_8c18_8557274b30e0.slice/crio-d9123e7d40814eaa99a6569845ec968603ae884fea3e5b290523b09e2f0f4dce WatchSource:0}: Error finding container d9123e7d40814eaa99a6569845ec968603ae884fea3e5b290523b09e2f0f4dce: Status 404 returned error can't find the container with id d9123e7d40814eaa99a6569845ec968603ae884fea3e5b290523b09e2f0f4dce
Nov 03 01:20:32.651408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.651379    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:20:32.653324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.653291    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:32.653396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.653361    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:32.653396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.653389    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:32.763211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.743570    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"]
Nov 03 01:20:32.775189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.772444    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:32.775189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.772524    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:32.775189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:32.772561    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:20:33.097116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.097018    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" event={"ID":"37b1042d-911e-4e83-b83d-dbb8096646d3","Type":"ContainerStarted","Data":"2813854ffe6ce87687485dfe388786010fe63381bed47f66672f41dbaaba0a93"}
Nov 03 01:20:33.097116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.097123    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" event={"ID":"37b1042d-911e-4e83-b83d-dbb8096646d3","Type":"ContainerStarted","Data":"bf24d0a6dad9fbf1fbb81e37d187ba5fe56edbbad9c1dbdde685322cc03a4dad"}
Nov 03 01:20:33.100549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.100514    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" event={"ID":"74d4dcef-ba0d-4f3b-8c18-8557274b30e0","Type":"ContainerStarted","Data":"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"}
Nov 03 01:20:33.100714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.100574    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" event={"ID":"74d4dcef-ba0d-4f3b-8c18-8557274b30e0","Type":"ContainerStarted","Data":"d9123e7d40814eaa99a6569845ec968603ae884fea3e5b290523b09e2f0f4dce"}
Nov 03 01:20:33.100845 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.100734    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"
Nov 03 01:20:33.126651 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.126576    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podStartSLOduration=2.126544556 podStartE2EDuration="2.126544556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:33.118002192 +0000 UTC m=+2378.331745869" watchObservedRunningTime="2026-11-03 01:20:33.126544556 +0000 UTC m=+2378.340288229"
Nov 03 01:20:33.160967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.160840    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podStartSLOduration=2.160819521 podStartE2EDuration="2.160819521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:33.145724156 +0000 UTC m=+2378.359468135" watchObservedRunningTime="2026-11-03 01:20:33.160819521 +0000 UTC m=+2378.374563198"
Nov 03 01:20:33.167762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.167723    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 01:20:33.175534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.175497    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-6gtmn"]
Nov 03 01:20:33.357209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.357155    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:33.358309 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.357641    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:33.358563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.358538    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:33.399953 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.399910    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:33.401893 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.401860    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:33.402040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:33.401923    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:34.367184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:34.365677    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:34.367184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:34.365744    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:34.399448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:34.399401    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:34.399631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:34.399469    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:34.975188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:34.972513    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" path="/var/lib/kubelet/pods/1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2/volumes"
Nov 03 01:20:35.360850 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:35.360804    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:35.361206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:35.361170    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:35.405294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:35.405256    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:35.405661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:35.405621    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:36.355619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:36.355571    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:36.362191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:36.358013    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:36.402500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:36.402009    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:36.402500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:36.402078    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:37.355627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.355572    4531 generic.go:356] "Generic (PLEG): container finished" podID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerID="b09fdcbd8568004d0b6ce438e7f1f39229ae45f1b77f01837050ad91bbd3dae4" exitCode=0
Nov 03 01:20:37.356745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.356716    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" event={"ID":"5940c844-7285-498e-8bae-47ecfe2eb48d","Type":"ContainerDied","Data":"b09fdcbd8568004d0b6ce438e7f1f39229ae45f1b77f01837050ad91bbd3dae4"}
Nov 03 01:20:37.382912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.382874    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:37.383642 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.383606    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:37.441186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.439821    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:37.441186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.439880    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:37.940223 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:37.934588    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:20:38.086717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.086676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.087086 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.087061    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.088332 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.087378    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.088616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.088592    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.088811 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.088794    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.088987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.088972    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.089103 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.089089    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.092377 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.092360    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.093564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.092596    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\") pod \"5940c844-7285-498e-8bae-47ecfe2eb48d\" (UID: \"5940c844-7285-498e-8bae-47ecfe2eb48d\") "
Nov 03 01:20:38.095306 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.095282    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:20:38.095745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.095723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:38.101565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.101532    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:38.110669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.110293    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:38.110669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.110383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:20:38.111436 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.111385    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:38.111868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.111818    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "kube-api-access-96m2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:38.113631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.113606    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:20:38.113747 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.113606    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy" pod "5940c844-7285-498e-8bae-47ecfe2eb48d" (UID: "5940c844-7285-498e-8bae-47ecfe2eb48d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:20:38.196095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196049    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196083    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196093    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196107    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196120    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5940c844-7285-498e-8bae-47ecfe2eb48d-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196128    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196136    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196144    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5940c844-7285-498e-8bae-47ecfe2eb48d-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.196381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.196153    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96m2b\" (UniqueName: \"kubernetes.io/projected/5940c844-7285-498e-8bae-47ecfe2eb48d-kube-api-access-96m2b\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:20:38.386240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.385298    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:38.386240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.385378    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:38.400188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.400093    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:38.400188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.400143    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:38.402851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.402825    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"
Nov 03 01:20:38.402977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.402879    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz" event={"ID":"5940c844-7285-498e-8bae-47ecfe2eb48d","Type":"ContainerDied","Data":"da8093a2518967937003785dcd781208da35bf8c186612f9d01c6849c6d61084"}
Nov 03 01:20:38.402977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.402934    4531 scope.go:122] "RemoveContainer" containerID="b09fdcbd8568004d0b6ce438e7f1f39229ae45f1b77f01837050ad91bbd3dae4"
Nov 03 01:20:38.575333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.575256    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:20:38.628190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.617590    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-w9dmz"]
Nov 03 01:20:38.991512 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:38.980295    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" path="/var/lib/kubelet/pods/5940c844-7285-498e-8bae-47ecfe2eb48d/volumes"
Nov 03 01:20:39.361185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:39.360815    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:39.361185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:39.360892    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:39.400068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:39.400016    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:39.400609 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:39.400088    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:39.988192 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:20:39.984506    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:20:40.394192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:40.388379    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:40.394192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:40.388444    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:40.400105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:40.400074    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:40.400591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:40.400566    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:41.354974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:41.354926    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:41.355310 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:41.355274    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:41.399823 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:41.399779    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:41.400017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:41.399856    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:42.356198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:42.355778    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 01:20:42.356198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:42.355852    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 01:20:42.403547 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:42.403510    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 03 01:20:42.403802 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:42.403777    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 03 01:20:44.472259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:44.472209    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:20:44.472829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:44.472782    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:20:44.473562 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:44.473530    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-fjz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:20:44.489212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:44.488001    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:20:45.257460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.257424    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:45.258903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.258741    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:45.259635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.259592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:20:45.306559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.306523    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:45.307445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.307417    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:45.308784 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:45.308763    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd"
Nov 03 01:20:56.158184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:20:56.135987    4531 scope.go:122] "RemoveContainer" containerID="08ea36b41483f1fc91bc3a0612c8d9f5190b53ce11d9a1b435a7be5a0361f72f"
Nov 03 01:21:09.892950 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:21:09.892908    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:21:34.803504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:34.803446    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:21:34.804243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:34.803801    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy" containerID="cri-o://4cafa7f0b8266d7c4433683e6371fe8fc1bad75ebc9a9e097b703d7f19df3066" gracePeriod=30
Nov 03 01:21:39.883184 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:21:39.883123    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:21:40.216265 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.216043    4531 generic.go:356] "Generic (PLEG): container finished" podID="9545e67c-f608-4686-be71-89cc29823e3e" containerID="4cafa7f0b8266d7c4433683e6371fe8fc1bad75ebc9a9e097b703d7f19df3066" exitCode=0
Nov 03 01:21:40.216265 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.216154    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" event={"ID":"9545e67c-f608-4686-be71-89cc29823e3e","Type":"ContainerDied","Data":"4cafa7f0b8266d7c4433683e6371fe8fc1bad75ebc9a9e097b703d7f19df3066"}
Nov 03 01:21:40.415865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.415836    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:21:40.522682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522573    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.522682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522655    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.522912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522779    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.522912 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522867    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523031 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522944    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523031 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.522988    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:21:40.523031 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523003    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523322 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523156    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523449    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523285    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:40.523563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523410    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:40.523563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523486    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:40.523563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523493    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\") pod \"9545e67c-f608-4686-be71-89cc29823e3e\" (UID: \"9545e67c-f608-4686-be71-89cc29823e3e\") "
Nov 03 01:21:40.523789 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.523728    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:40.524274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.524253    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.524274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.524276    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.524466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.524294    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9545e67c-f608-4686-be71-89cc29823e3e-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.524466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.524308    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.524466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.524320    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.526136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.526108    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:40.526803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.526651    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:21:40.526803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.526706    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "kube-api-access-ctvpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:40.526803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.526785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token" pod "9545e67c-f608-4686-be71-89cc29823e3e" (UID: "9545e67c-f608-4686-be71-89cc29823e3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:40.809935 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.797857    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvpq\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-kube-api-access-ctvpq\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.809935 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.797897    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9545e67c-f608-4686-be71-89cc29823e3e-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.809935 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.797914    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9545e67c-f608-4686-be71-89cc29823e3e-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:40.809935 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:40.797929    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9545e67c-f608-4686-be71-89cc29823e3e-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:41.343232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:41.332719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2" event={"ID":"9545e67c-f608-4686-be71-89cc29823e3e","Type":"ContainerDied","Data":"487517a1f55d1cc8d7c70e16c61e3139e35a9ab04f5392527b93c29e18b19f71"}
Nov 03 01:21:41.343232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:41.332827    4531 scope.go:122] "RemoveContainer" containerID="4cafa7f0b8266d7c4433683e6371fe8fc1bad75ebc9a9e097b703d7f19df3066"
Nov 03 01:21:41.343232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:41.333081    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"
Nov 03 01:21:41.380998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:41.380903    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:21:41.395025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:41.392743    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-k65g2"]
Nov 03 01:21:42.905002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:42.904957    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9545e67c-f608-4686-be71-89cc29823e3e" path="/var/lib/kubelet/pods/9545e67c-f608-4686-be71-89cc29823e3e/volumes"
Nov 03 01:21:46.508316 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:46.508272    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:21:46.520081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:46.520038    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy" containerID="cri-o://d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404" gracePeriod=30
Nov 03 01:21:51.860540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.860499    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:21:51.967177 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967123    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967204    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967242    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967433    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967620    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967741    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.967853 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.967787    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\") pod \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\" (UID: \"e5c2c1d6-30ac-4b37-993d-e667f98d6e43\") "
Nov 03 01:21:51.969019 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.968986    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:51.970567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.970542    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:51.972601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.972578    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:21:51.976885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.976841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:51.977304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.977112    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:21:51.977411 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.977302    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:51.984401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.984354    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:21:51.984739 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.984710    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:51.985728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:51.985700    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc" pod "e5c2c1d6-30ac-4b37-993d-e667f98d6e43" (UID: "e5c2c1d6-30ac-4b37-993d-e667f98d6e43"). InnerVolumeSpecName "kube-api-access-v4nzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:21:52.069840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069801    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.069840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069835    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069856    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069867    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nzc\" (UniqueName: \"kubernetes.io/projected/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-kube-api-access-v4nzc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069878    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069887    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069896    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069904    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.070060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.069912    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5c2c1d6-30ac-4b37-993d-e667f98d6e43-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:21:52.489987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.489953    4531 generic.go:356] "Generic (PLEG): container finished" podID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerID="d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404" exitCode=0
Nov 03 01:21:52.490629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.490582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" event={"ID":"e5c2c1d6-30ac-4b37-993d-e667f98d6e43","Type":"ContainerDied","Data":"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"}
Nov 03 01:21:52.490871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.490843    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w" event={"ID":"e5c2c1d6-30ac-4b37-993d-e667f98d6e43","Type":"ContainerDied","Data":"5c41e3ef47491eefe1d044c39a7a563fade87632a41400a1779a71ae4e02882e"}
Nov 03 01:21:52.491136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.491115    4531 scope.go:122] "RemoveContainer" containerID="d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"
Nov 03 01:21:52.491427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.491411    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"
Nov 03 01:21:52.628074 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.628042    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:21:52.628625 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.628608    4531 scope.go:122] "RemoveContainer" containerID="d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"
Nov 03 01:21:52.629482 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:21:52.629339    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404\": container with ID starting with d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404 not found: ID does not exist" containerID="d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"
Nov 03 01:21:52.629482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.629383    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404"} err="failed to get container status \"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404\": rpc error: code = NotFound desc = could not find container \"d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404\": container with ID starting with d8b2d1119e2ca2d4456908d880dd8fa2f4edbf26a9ad5bdf5b1ea1510d15c404 not found: ID does not exist"
Nov 03 01:21:52.647193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.640967    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-jl44w"]
Nov 03 01:21:52.906256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:21:52.906215    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" path="/var/lib/kubelet/pods/e5c2c1d6-30ac-4b37-993d-e667f98d6e43/volumes"
Nov 03 01:22:09.881086 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:22:09.881005    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:22:22.955985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.955938    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:22:22.957083 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957062    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy"
Nov 03 01:22:22.957129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957087    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy"
Nov 03 01:22:22.957129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957110    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy"
Nov 03 01:22:22.957129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957120    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy"
Nov 03 01:22:22.957241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957137    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy"
Nov 03 01:22:22.957241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957145    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy"
Nov 03 01:22:22.957241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957182    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy"
Nov 03 01:22:22.957241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957191    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy"
Nov 03 01:22:22.957392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957380    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4cb3e6-2e46-43c9-82f5-7ebc9569b3f2" containerName="istio-proxy"
Nov 03 01:22:22.957426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957397    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c2c1d6-30ac-4b37-993d-e667f98d6e43" containerName="istio-proxy"
Nov 03 01:22:22.957426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957407    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5940c844-7285-498e-8bae-47ecfe2eb48d" containerName="istio-proxy"
Nov 03 01:22:22.957426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.957418    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9545e67c-f608-4686-be71-89cc29823e3e" containerName="istio-proxy"
Nov 03 01:22:22.962042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.962018    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:22.966307 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.966282    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-821f5a-openshift-default-dockercfg-8ft6g\""
Nov 03 01:22:22.966746 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:22.966719    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:22:23.117565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117473    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.117942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117677    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thscj\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.117942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117720    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.117942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117835    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.117942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117866    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.118256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.117998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.118256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.118042    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.118256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.118102    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.118256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.118141    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220416    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220719 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220952 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220810    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.220952 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220948    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.220963    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thscj\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.221008    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.221137    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221358 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.221331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.221439 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.221364    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.223027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.222998    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.223271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.223254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.228441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.228416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.228561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.228455    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thscj\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\") pod \"e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") " pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.287218 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.287175    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:23.580574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.580541    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:22:23.581675 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:22:23.581648    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21dd413_d51f_4eaf_9f47_12eab1686a34.slice/crio-30bad4ba448ed039913d63ec1467120bba22e6d08f8d3d033b8aff7afab58359 WatchSource:0}: Error finding container 30bad4ba448ed039913d63ec1467120bba22e6d08f8d3d033b8aff7afab58359: Status 404 returned error can't find the container with id 30bad4ba448ed039913d63ec1467120bba22e6d08f8d3d033b8aff7afab58359
Nov 03 01:22:23.585253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.585219    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:23.585340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.585288    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:23.585340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:23.585319    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:22:24.134180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.133696    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" event={"ID":"f21dd413-d51f-4eaf-9f47-12eab1686a34","Type":"ContainerStarted","Data":"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"}
Nov 03 01:22:24.134180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.133768    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" event={"ID":"f21dd413-d51f-4eaf-9f47-12eab1686a34","Type":"ContainerStarted","Data":"30bad4ba448ed039913d63ec1467120bba22e6d08f8d3d033b8aff7afab58359"}
Nov 03 01:22:24.156135 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.156066    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podStartSLOduration=2.156046703 podStartE2EDuration="2.156046703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:24.153138694 +0000 UTC m=+2489.366882368" watchObservedRunningTime="2026-11-03 01:22:24.156046703 +0000 UTC m=+2489.369790376"
Nov 03 01:22:24.287590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.287555    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:24.289074 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.289045    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:24.289255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:24.289101    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:25.296385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:25.291527    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:25.296385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:25.291599    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:26.288558 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:26.288519    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:26.288757 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:26.288575    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:27.298820 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:27.298503    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:27.298820 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:27.298563    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:28.288094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:28.288055    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:28.288317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:28.288126    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:29.287943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:29.287905    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:29.288380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:29.287977    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:30.288133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:30.288090    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 01:22:30.288586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:30.288154    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 01:22:31.294954 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:31.294921    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:31.295638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:31.295352    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:31.296127 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:31.296104    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:22:39.882792 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:22:39.882748    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:22:45.607986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.607940    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:22:45.625460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.625408    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:22:45.625737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.625528    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.630806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.630764    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a88862-serving-cert\""
Nov 03 01:22:45.751011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.750967    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.751328 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.751298    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.751765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.751732    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96x6f\" (UniqueName: \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.755932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.752098    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.858104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.857992    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.858104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.858038    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.858400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.858182    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96x6f\" (UniqueName: \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.858400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.858351    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.858906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.858875    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.860787 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.860764    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.860915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.860888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.866127 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.866103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96x6f\" (UniqueName: \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\") pod \"maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:45.983756 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:45.983715    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:46.408192 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:46.406615    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:22:46.467199 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:46.450832    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:22:46.755703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:46.755588    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" event={"ID":"7094882d-8427-4d75-ac29-73e788cd33c0","Type":"ContainerStarted","Data":"1578028b7f387af7b9f061fec35477725d18296a3f6edb44bfe3f5e06d36fff4"}
Nov 03 01:22:47.766949 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:47.766896    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" event={"ID":"7094882d-8427-4d75-ac29-73e788cd33c0","Type":"ContainerStarted","Data":"84902bef4eaeab3451b9c8ae153284123f233367dd4b7bc188fc5f6f9719b323"}
Nov 03 01:22:47.767463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:47.767012    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:22:47.783999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:47.783932    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" podStartSLOduration=2.256660703 podStartE2EDuration="2.783900194s" totalImagesPullingTime="527.239491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:47.78363398 +0000 UTC m=+2512.997377655" watchObservedRunningTime="2026-11-03 01:22:47.783900194 +0000 UTC m=+2512.997643867"
Nov 03 01:22:53.782396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:22:53.782365    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:23:09.892151 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:23:09.892114    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:23:34.222151 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.222111    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:23:34.237564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.237537    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.239154 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.239124    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:23:34.244286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.244252    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ce04af-serving-cert\""
Nov 03 01:23:34.381185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.364729    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.381550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.381519    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.389345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.389306    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.389841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.389814    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k54c\" (UniqueName: \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.531556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.531459    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.531743 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.531600    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.531810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.531747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k54c\" (UniqueName: \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.531908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.531890    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.532336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.532314    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.534325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.534301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.534537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.534506    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.540844 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.540814    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k54c\" (UniqueName: \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\") pod \"maas-api-e2e-inf-b-ce04af-554767746d-z5fdm\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.565675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:34.565640    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:34.978187 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:23:34.961524    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716fa21a_6f33_4ee3_99d1_a24e2062d671.slice/crio-55859b3a9e3093d16af2453dbb5c2fd24cb5e765b14f60db3688b2a2c3a23192 WatchSource:0}: Error finding container 55859b3a9e3093d16af2453dbb5c2fd24cb5e765b14f60db3688b2a2c3a23192: Status 404 returned error can't find the container with id 55859b3a9e3093d16af2453dbb5c2fd24cb5e765b14f60db3688b2a2c3a23192
Nov 03 01:23:35.017184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:35.014370    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:23:35.450940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:35.450908    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" event={"ID":"716fa21a-6f33-4ee3-99d1-a24e2062d671","Type":"ContainerStarted","Data":"55859b3a9e3093d16af2453dbb5c2fd24cb5e765b14f60db3688b2a2c3a23192"}
Nov 03 01:23:36.461617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:36.461578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" event={"ID":"716fa21a-6f33-4ee3-99d1-a24e2062d671","Type":"ContainerStarted","Data":"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"}
Nov 03 01:23:36.462116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:36.461659    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:23:36.483766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:36.483711    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" podStartSLOduration=2.047605853 podStartE2EDuration="2.48369684s" totalImagesPullingTime="436.090987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:36.481115611 +0000 UTC m=+2561.694859283" watchObservedRunningTime="2026-11-03 01:23:36.48369684 +0000 UTC m=+2561.697440575"
Nov 03 01:23:39.880237 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:23:39.880181    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:23:42.477408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:23:42.477376    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:24:09.938527 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:24:09.938479    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:24:14.401182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.386220    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:14.439188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.428581    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.555215 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.547960    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"openshift-service-ca.crt\""
Nov 03 01:24:14.569119 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.569087    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-758025cf8245\""
Nov 03 01:24:14.570355 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.570332    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-22a51f875964\""
Nov 03 01:24:14.570581 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.570567    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"agc-cred-8aa58f91c6d5\""
Nov 03 01:24:14.570783 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.570767    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-e2e--4d491765-dockercfg-56tgf\""
Nov 03 01:24:14.570978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.570964    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"routing-overlay\""
Nov 03 01:24:14.571217 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.571201    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"kube-root-ca.crt\""
Nov 03 01:24:14.571410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.571392    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d25759\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d25759\""
Nov 03 01:24:14.640016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.639980    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:14.674142 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.671581    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.674693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.674664    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h848\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.675002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.674986    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.675414 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.675394    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.777233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.777184    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.777457 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.777338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.777457 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.777407    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h848\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.777578 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.777487    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.778089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.778055    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.778089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.778082    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.780574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.780544    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.785311 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.785288    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h848\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\") pod \"payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") " pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.832926 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.832882    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:14.840901 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.840868    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.844795    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"agc-cred-4646824b7cb1\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.844803    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"agc-cred-fbc8c941c1bc\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.844854    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"agc-cred-f049cc72059c\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.844992    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"payload-processing-external-model-e2e--b70757fa-dockercfg-s4rrb\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.845344    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"kube-root-ca.crt\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.845479    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"openshift-service-ca.crt\""
Nov 03 01:24:14.845561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.845524    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e1f08d\""
Nov 03 01:24:14.846038 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.845742    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:14.846288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.846262    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e1f08d\"/\"routing-overlay\""
Nov 03 01:24:14.879000 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.878961    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.879197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.879043    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.879256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.879205    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vqj\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.879385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.879360    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.950271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.950121    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:14.981524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.981490    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vqj\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.981699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.981680    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.982485 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.982460    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.982566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.982549    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.982654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.982631    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.983409 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.983380    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.986097 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.986073    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:14.992740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:14.992716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vqj\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\") pod \"payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") " pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:15.175427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.175388    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:15.318555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.318502    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:15.470605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.469787    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:15.473878 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:24:15.473833    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82da90ea_0737_454b_a5b1_728d0b6b1829.slice/crio-1ec1db89694cdd71aaf0fbd1a0772c9fb734ca3b8c59829e62c995a92ac42406 WatchSource:0}: Error finding container 1ec1db89694cdd71aaf0fbd1a0772c9fb734ca3b8c59829e62c995a92ac42406: Status 404 returned error can't find the container with id 1ec1db89694cdd71aaf0fbd1a0772c9fb734ca3b8c59829e62c995a92ac42406
Nov 03 01:24:15.983581 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.983532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" event={"ID":"82da90ea-0737-454b-a5b1-728d0b6b1829","Type":"ContainerStarted","Data":"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"}
Nov 03 01:24:15.983789 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.983675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" event={"ID":"82da90ea-0737-454b-a5b1-728d0b6b1829","Type":"ContainerStarted","Data":"1ec1db89694cdd71aaf0fbd1a0772c9fb734ca3b8c59829e62c995a92ac42406"}
Nov 03 01:24:15.983789 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.983751    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:15.990232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.989360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" event={"ID":"e0832029-dab3-48d3-862a-25843e9c31ea","Type":"ContainerStarted","Data":"5fe75a69c505afebaaa30c370f6e62811d52fa3af6d25d2cb0170e1545fd3222"}
Nov 03 01:24:15.990232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.989438    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" event={"ID":"e0832029-dab3-48d3-862a-25843e9c31ea","Type":"ContainerStarted","Data":"2b3568a257e5216a3e4ae3ef7b63dbf0e14d93619d5d8dae59285ff16a1334a6"}
Nov 03 01:24:15.990232 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:15.989477    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:16.004738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.004675    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" podStartSLOduration=2.004656451 podStartE2EDuration="2.004656451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:15.999743861 +0000 UTC m=+2601.213487534" watchObservedRunningTime="2026-11-03 01:24:16.004656451 +0000 UTC m=+2601.218400124"
Nov 03 01:24:16.020234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.018323    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" podStartSLOduration=2.018308045 podStartE2EDuration="2.018308045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:16.017013632 +0000 UTC m=+2601.230757306" watchObservedRunningTime="2026-11-03 01:24:16.018308045 +0000 UTC m=+2601.232051718"
Nov 03 01:24:16.223294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.218373    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:24:16.240951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.240865    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.253527    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"openshift-service-ca.crt\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254310    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"routing-overlay\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254379    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-7c759a\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254455    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"kube-root-ca.crt\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254559    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"payload-processing-external-model-e2e--d95606a1-dockercfg-dl4kq\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254622    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"agc-cred-2c199a80818f\""
Nov 03 01:24:16.256602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.254702    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-7c759a\"/\"agc-cred-be1d6e8d29d8\""
Nov 03 01:24:16.274965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.268734    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:24:16.445844 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.445782    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.446308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.446275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28t4\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.446932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.446403    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.446932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.446486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.548792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.548687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.549616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.549504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.551076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.551049    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28t4\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.551253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.551138    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.551253 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.551242    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.552120 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.552096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.561211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.561183    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28t4\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.564426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.563267    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") " pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:16.719186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:16.701322    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:17.450777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:17.450742    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:24:17.452670 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:24:17.452638    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7f9f60_3ee0_4e6a_ab1f_f273969b9687.slice/crio-0e940f2c71b9ea357a81414112abe3cf9614385f998d194b2d2e8387d871529f WatchSource:0}: Error finding container 0e940f2c71b9ea357a81414112abe3cf9614385f998d194b2d2e8387d871529f: Status 404 returned error can't find the container with id 0e940f2c71b9ea357a81414112abe3cf9614385f998d194b2d2e8387d871529f
Nov 03 01:24:18.150993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.150952    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:18.255702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.255662    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" event={"ID":"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687","Type":"ContainerStarted","Data":"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"}
Nov 03 01:24:18.255903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.255729    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" event={"ID":"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687","Type":"ContainerStarted","Data":"0e940f2c71b9ea357a81414112abe3cf9614385f998d194b2d2e8387d871529f"}
Nov 03 01:24:18.255903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.255760    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:18.272827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.272773    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" podStartSLOduration=2.272759846 podStartE2EDuration="2.272759846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:18.270554364 +0000 UTC m=+2603.484298032" watchObservedRunningTime="2026-11-03 01:24:18.272759846 +0000 UTC m=+2603.486503592"
Nov 03 01:24:18.649395 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.646318    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:24:18.652662 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.652633    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.652900 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.652874    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:24:18.656021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.655999    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-d1575a046604\""
Nov 03 01:24:18.656696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.656676    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"agc-cred-a6687acf616c\""
Nov 03 01:24:18.656945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.656930    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-e2e--4a3ed774-dockercfg-5lwqz\""
Nov 03 01:24:18.657470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.657360    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"routing-overlay\""
Nov 03 01:24:18.657671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.657655    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"openshift-service-ca.crt\""
Nov 03 01:24:18.658817 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.658801    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-57079b\""
Nov 03 01:24:18.660350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.660332    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-57079b\"/\"kube-root-ca.crt\""
Nov 03 01:24:18.811026 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.810987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.811304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.811274    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.811384 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.811325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h89\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.811534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.811516    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.913412 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.913324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.913572 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.913499    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.913572 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.913525    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h89\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.913744 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.913726    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.914088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.914063    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.914368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.914345    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.916122 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.916102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.921525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.921503    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h89\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\") pod \"payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") " pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:18.984496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:18.984457    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:19.024112 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:19.024068    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:19.276438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:19.276259    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:24:20.278382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:20.278340    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" event={"ID":"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed","Type":"ContainerStarted","Data":"3c699ead40280b2082c951483ef7e22bf396b18a385b936ac627d76a2facb207"}
Nov 03 01:24:20.278911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:20.278408    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" event={"ID":"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed","Type":"ContainerStarted","Data":"c70481cd00a6d2a670bf131edfa935b18f87586aba6f205176ffbd866f88d8b7"}
Nov 03 01:24:20.278911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:20.278431    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:20.294660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:20.294595    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" podStartSLOduration=2.294575769 podStartE2EDuration="2.294575769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:20.29379042 +0000 UTC m=+2605.507534094" watchObservedRunningTime="2026-11-03 01:24:20.294575769 +0000 UTC m=+2605.508319442"
Nov 03 01:24:23.256625 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.256544    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" podUID="82da90ea-0737-454b-a5b1-728d0b6b1829" containerName="payload-processing" containerID="cri-o://3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569" gracePeriod=25
Nov 03 01:24:23.560025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.559997    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:23.714721 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.713906    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\") pod \"82da90ea-0737-454b-a5b1-728d0b6b1829\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") "
Nov 03 01:24:23.715379 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.715330    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\") pod \"82da90ea-0737-454b-a5b1-728d0b6b1829\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") "
Nov 03 01:24:23.716106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.716084    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\") pod \"82da90ea-0737-454b-a5b1-728d0b6b1829\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") "
Nov 03 01:24:23.716697 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.716676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\") pod \"82da90ea-0737-454b-a5b1-728d0b6b1829\" (UID: \"82da90ea-0737-454b-a5b1-728d0b6b1829\") "
Nov 03 01:24:23.717247 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.715707    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume" pod "82da90ea-0737-454b-a5b1-728d0b6b1829" (UID: "82da90ea-0737-454b-a5b1-728d0b6b1829"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:23.717407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.717207    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume" pod "82da90ea-0737-454b-a5b1-728d0b6b1829" (UID: "82da90ea-0737-454b-a5b1-728d0b6b1829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:23.718407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.718381    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:23.718617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.718597    4531 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82da90ea-0737-454b-a5b1-728d0b6b1829-routing-overlay-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:23.725430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.725214    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj" pod "82da90ea-0737-454b-a5b1-728d0b6b1829" (UID: "82da90ea-0737-454b-a5b1-728d0b6b1829"). InnerVolumeSpecName "kube-api-access-b8vqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:23.732525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.732488    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials" pod "82da90ea-0737-454b-a5b1-728d0b6b1829" (UID: "82da90ea-0737-454b-a5b1-728d0b6b1829"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:23.820658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.820625    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vqj\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-kube-api-access-b8vqj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:23.820658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:23.820660    4531 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82da90ea-0737-454b-a5b1-728d0b6b1829-provider-credentials\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:24.025289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.025238    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" podUID="e0832029-dab3-48d3-862a-25843e9c31ea" containerName="payload-processing" containerID="cri-o://5fe75a69c505afebaaa30c370f6e62811d52fa3af6d25d2cb0170e1545fd3222" gracePeriod=25
Nov 03 01:24:24.363233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.360087    4531 generic.go:356] "Generic (PLEG): container finished" podID="82da90ea-0737-454b-a5b1-728d0b6b1829" containerID="3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569" exitCode=0
Nov 03 01:24:24.368256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.364370    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" event={"ID":"82da90ea-0737-454b-a5b1-728d0b6b1829","Type":"ContainerDied","Data":"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"}
Nov 03 01:24:24.368256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.364447    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4" event={"ID":"82da90ea-0737-454b-a5b1-728d0b6b1829","Type":"ContainerDied","Data":"1ec1db89694cdd71aaf0fbd1a0772c9fb734ca3b8c59829e62c995a92ac42406"}
Nov 03 01:24:24.368256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.364477    4531 scope.go:122] "RemoveContainer" containerID="3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"
Nov 03 01:24:24.368256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.364680    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"
Nov 03 01:24:24.405560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.405518    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0832029-dab3-48d3-862a-25843e9c31ea" containerID="5fe75a69c505afebaaa30c370f6e62811d52fa3af6d25d2cb0170e1545fd3222" exitCode=0
Nov 03 01:24:24.405698 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.405603    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" event={"ID":"e0832029-dab3-48d3-862a-25843e9c31ea","Type":"ContainerDied","Data":"5fe75a69c505afebaaa30c370f6e62811d52fa3af6d25d2cb0170e1545fd3222"}
Nov 03 01:24:24.405698 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.405656    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl" event={"ID":"e0832029-dab3-48d3-862a-25843e9c31ea","Type":"ContainerDied","Data":"2b3568a257e5216a3e4ae3ef7b63dbf0e14d93619d5d8dae59285ff16a1334a6"}
Nov 03 01:24:24.405698 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.405683    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3568a257e5216a3e4ae3ef7b63dbf0e14d93619d5d8dae59285ff16a1334a6"
Nov 03 01:24:24.408313 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.408286    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:24.414130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.414103    4531 scope.go:122] "RemoveContainer" containerID="3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"
Nov 03 01:24:24.414530 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:24:24.414501    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569\": container with ID starting with 3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569 not found: ID does not exist" containerID="3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"
Nov 03 01:24:24.414619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.414538    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569"} err="failed to get container status \"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569\": rpc error: code = NotFound desc = could not find container \"3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569\": container with ID starting with 3693b97b597d20b27412f5e830a2a57968d4f15c4697d9033f6a549eefe75569 not found: ID does not exist"
Nov 03 01:24:24.422608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.422577    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:24.427561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.427531    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e1f08d/payload-processing-external-model-e2e-extmod-b-e1f08d-75c8wzcv4"]
Nov 03 01:24:24.447337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.447297    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\") pod \"e0832029-dab3-48d3-862a-25843e9c31ea\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") "
Nov 03 01:24:24.447603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.447570    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\") pod \"e0832029-dab3-48d3-862a-25843e9c31ea\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") "
Nov 03 01:24:24.448015 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.447993    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\") pod \"e0832029-dab3-48d3-862a-25843e9c31ea\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") "
Nov 03 01:24:24.448134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.448016    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume" pod "e0832029-dab3-48d3-862a-25843e9c31ea" (UID: "e0832029-dab3-48d3-862a-25843e9c31ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:24.448134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.448032    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\") pod \"e0832029-dab3-48d3-862a-25843e9c31ea\" (UID: \"e0832029-dab3-48d3-862a-25843e9c31ea\") "
Nov 03 01:24:24.448710 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.448687    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume" pod "e0832029-dab3-48d3-862a-25843e9c31ea" (UID: "e0832029-dab3-48d3-862a-25843e9c31ea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:24.448839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.448814    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:24.456012 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.455982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials" pod "e0832029-dab3-48d3-862a-25843e9c31ea" (UID: "e0832029-dab3-48d3-862a-25843e9c31ea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:24.457392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.457365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848" pod "e0832029-dab3-48d3-862a-25843e9c31ea" (UID: "e0832029-dab3-48d3-862a-25843e9c31ea"). InnerVolumeSpecName "kube-api-access-8h848". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:24.555412 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.555372    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h848\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-kube-api-access-8h848\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:24.555832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.555692    4531 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0832029-dab3-48d3-862a-25843e9c31ea-routing-overlay-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:24.556043 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.556024    4531 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0832029-dab3-48d3-862a-25843e9c31ea-provider-credentials\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:24:24.911359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:24.911314    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82da90ea-0737-454b-a5b1-728d0b6b1829" path="/var/lib/kubelet/pods/82da90ea-0737-454b-a5b1-728d0b6b1829/volumes"
Nov 03 01:24:25.463185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.462731    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"
Nov 03 01:24:25.554185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.553225    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:25.610201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.601209    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d25759/payload-processing-external-model-e2e-extmod-a-d25759-57c8rw5kl"]
Nov 03 01:24:25.625238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.625121    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626565    4531 cpu_manager.go:414] "containerMap: removing container" podUID="82da90ea-0737-454b-a5b1-728d0b6b1829" containerName="payload-processing"
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626594    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82da90ea-0737-454b-a5b1-728d0b6b1829" containerName="payload-processing"
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626667    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e0832029-dab3-48d3-862a-25843e9c31ea" containerName="payload-processing"
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626675    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0832029-dab3-48d3-862a-25843e9c31ea" containerName="payload-processing"
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626827    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0832029-dab3-48d3-862a-25843e9c31ea" containerName="payload-processing"
Nov 03 01:24:25.640293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.626851    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="82da90ea-0737-454b-a5b1-728d0b6b1829" containerName="payload-processing"
Nov 03 01:24:25.666307 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.666125    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.672001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.671557    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce04af-openshift-default-dockercfg-rdq4s\""
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.703969    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704096    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vbl\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704185    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704269    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704319    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704489    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704603    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.727305 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.704649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.783638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.779106    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:24:25.806276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.806236    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vbl\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.806464 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.806320    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.806464 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.806372    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.806464 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.806405    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.806657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.806473    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.807526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.807499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.811575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.811524    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.811732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.811715    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.811818 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.811787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.811971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.811943    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.813926 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.813084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.816069 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.816028    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.816647 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.816614    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.819736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.819678    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.824591 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.824560    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.840788 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.839593    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vbl\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.840788 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.840688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:25.847495 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:25.844677    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\") pod \"e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") " pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:26.098085 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.098048    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:26.396927 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:24:26.396894    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac858873_189e_4f6a_b104_3c698fe7c16a.slice/crio-79a3dab8f9b1fbf4bb389760bc83cacdbffa207031789d1c16b6106d78a1afde WatchSource:0}: Error finding container 79a3dab8f9b1fbf4bb389760bc83cacdbffa207031789d1c16b6106d78a1afde: Status 404 returned error can't find the container with id 79a3dab8f9b1fbf4bb389760bc83cacdbffa207031789d1c16b6106d78a1afde
Nov 03 01:24:26.397514 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.397486    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:24:26.399950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.399914    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:24:26.400048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.399984    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:24:26.400048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.400014    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:24:26.496096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.496054    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" event={"ID":"ac858873-189e-4f6a-b104-3c698fe7c16a","Type":"ContainerStarted","Data":"79a3dab8f9b1fbf4bb389760bc83cacdbffa207031789d1c16b6106d78a1afde"}
Nov 03 01:24:26.906407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:26.906372    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0832029-dab3-48d3-862a-25843e9c31ea" path="/var/lib/kubelet/pods/e0832029-dab3-48d3-862a-25843e9c31ea/volumes"
Nov 03 01:24:27.505223 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:27.505182    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" event={"ID":"ac858873-189e-4f6a-b104-3c698fe7c16a","Type":"ContainerStarted","Data":"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"}
Nov 03 01:24:27.551183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:27.550147    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podStartSLOduration=2.550123202 podStartE2EDuration="2.550123202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:27.523529017 +0000 UTC m=+2612.737272690" watchObservedRunningTime="2026-11-03 01:24:27.550123202 +0000 UTC m=+2612.763866875"
Nov 03 01:24:28.098419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:28.098383    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:28.100669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:28.100636    4531 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 01:24:28.100819 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:28.100705    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 01:24:29.099058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:29.099008    4531 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 01:24:29.099785 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:29.099747    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 01:24:29.269011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:29.268972    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:24:30.098868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:30.098826    4531 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 01:24:30.099042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:30.098917    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 01:24:31.098780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:31.098746    4531 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 01:24:31.099261 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:31.098801    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 01:24:31.290300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:31.290267    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:24:32.098810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:32.098768    4531 patch_prober.go:29] interesting pod/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 01:24:32.099249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:32.098827    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 01:24:33.117190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:33.113568    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:33.117190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:33.115688    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:33.126522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:24:33.126491    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:24:39.879546 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:24:39.879503    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:25:09.883930 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:25:09.883842    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:25:39.880283 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:25:39.880240    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:25:48.742443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:48.742400    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 01:25:48.743372 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:48.743312    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" containerID="cri-o://8f58813b0dd7b8e6fce14a3452b832be7b529a0d140683348eb62b32165e11f0" gracePeriod=30
Nov 03 01:25:53.137396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:53.137358    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:25:53.138023 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:53.137986    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:25:54.026875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.026834    4531 generic.go:356] "Generic (PLEG): container finished" podID="844f57de-e831-44bf-b3a5-f9532133f011" containerID="8f58813b0dd7b8e6fce14a3452b832be7b529a0d140683348eb62b32165e11f0" exitCode=0
Nov 03 01:25:54.027088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.026884    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" event={"ID":"844f57de-e831-44bf-b3a5-f9532133f011","Type":"ContainerDied","Data":"8f58813b0dd7b8e6fce14a3452b832be7b529a0d140683348eb62b32165e11f0"}
Nov 03 01:25:54.221815 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.221476    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 01:25:54.261522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.261483    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:25:54.262883 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.262855    4531 cpu_manager.go:414] "containerMap: removing container" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy"
Nov 03 01:25:54.262883 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.262880    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy"
Nov 03 01:25:54.263090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.263056    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="844f57de-e831-44bf-b3a5-f9532133f011" containerName="istio-proxy"
Nov 03 01:25:54.270929 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.270886    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.276660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.276023    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ee618c-serving-cert\""
Nov 03 01:25:54.286762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.286730    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:25:54.302858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.302827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.302894    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.302957    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.302988    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303028    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303098    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303132    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303281    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:54.303415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303367    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:54.303415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303398    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303436    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\") pod \"844f57de-e831-44bf-b3a5-f9532133f011\" (UID: \"844f57de-e831-44bf-b3a5-f9532133f011\") "
Nov 03 01:25:54.303524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303395    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:25:54.303635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303576    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:54.303733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303710    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:54.303911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303891    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2xn\" (UniqueName: \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.303992 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.303973    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.304141 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304121    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.304304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304280    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.304466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304448    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844f57de-e831-44bf-b3a5-f9532133f011-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.304538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304471    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.304538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304487    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.304538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304502    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.304538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.304517    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.305990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.305952    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "kube-api-access-pfl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:54.317684 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.317647    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:25:54.317829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.317718    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:54.320326 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.320300    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy" pod "844f57de-e831-44bf-b3a5-f9532133f011" (UID: "844f57de-e831-44bf-b3a5-f9532133f011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:54.409878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.406261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2xn\" (UniqueName: \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.409878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.406689    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.409878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.408550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.409878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.409502    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.409878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.409725    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.410483 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.410459    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844f57de-e831-44bf-b3a5-f9532133f011-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.410569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.410486    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.410569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.410500    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844f57de-e831-44bf-b3a5-f9532133f011-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.410569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.410514    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfl5c\" (UniqueName: \"kubernetes.io/projected/844f57de-e831-44bf-b3a5-f9532133f011-kube-api-access-pfl5c\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:25:54.410569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.410522    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.416038 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.416007    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.420942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.420899    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2xn\" (UniqueName: \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\") pod \"maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.594672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.594634    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:54.921010 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:54.920977    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:25:55.061301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.061243    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" event={"ID":"7f384fe7-0d42-437e-b037-464d5da5ae86","Type":"ContainerStarted","Data":"0de83ab9904ec7158bac9a331d7d3f153223ab8d6b3c8c7a152b0fe1148787a2"}
Nov 03 01:25:55.064757 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.064726    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd" event={"ID":"844f57de-e831-44bf-b3a5-f9532133f011","Type":"ContainerDied","Data":"4e32a3d6bef44577489270bb47221699947bb2b5d6a94221c1694bb082c310b9"}
Nov 03 01:25:55.065028 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.064986    4531 scope.go:122] "RemoveContainer" containerID="8f58813b0dd7b8e6fce14a3452b832be7b529a0d140683348eb62b32165e11f0"
Nov 03 01:25:55.065266 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.065251    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"
Nov 03 01:25:55.103213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.100274    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 01:25:55.144994 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:55.144945    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-g87sd"]
Nov 03 01:25:56.098986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:56.098877    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" event={"ID":"7f384fe7-0d42-437e-b037-464d5da5ae86","Type":"ContainerStarted","Data":"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"}
Nov 03 01:25:56.107187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:56.100193    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:25:56.162198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:56.148829    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" podStartSLOduration=38.57494422 podStartE2EDuration="39.148805127s" totalImagesPullingTime="573.860907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:56.134501086 +0000 UTC m=+2701.348244759" watchObservedRunningTime="2026-11-03 01:25:56.148805127 +0000 UTC m=+2701.362548800"
Nov 03 01:25:56.930537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:25:56.930498    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844f57de-e831-44bf-b3a5-f9532133f011" path="/var/lib/kubelet/pods/844f57de-e831-44bf-b3a5-f9532133f011/volumes"
Nov 03 01:26:03.150291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:03.150261    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:26:09.896831 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:26:09.896798    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:26:39.881288 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:26:39.881208    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:26:46.059182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:46.058915    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:26:46.672471 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:46.672415    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:26:51.065184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.064245    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" podUID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" containerName="payload-processing" containerID="cri-o://af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204" gracePeriod=25
Nov 03 01:26:51.389951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.389925    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:26:51.408913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.408876    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\") pod \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") "
Nov 03 01:26:51.409109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.408953    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\") pod \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") "
Nov 03 01:26:51.409109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.409050    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\") pod \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") "
Nov 03 01:26:51.409418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.409394    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\") pod \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\" (UID: \"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687\") "
Nov 03 01:26:51.410569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.410533    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume" pod "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" (UID: "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:51.411256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.411231    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume" pod "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" (UID: "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:51.416786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.416760    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials" pod "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" (UID: "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:51.417919 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.417894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4" pod "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" (UID: "2f7f9f60-3ee0-4e6a-ab1f-f273969b9687"). InnerVolumeSpecName "kube-api-access-f28t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:51.424327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.424295    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:26:51.425479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.425457    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" containerName="payload-processing"
Nov 03 01:26:51.425566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.425479    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" containerName="payload-processing"
Nov 03 01:26:51.425699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.425687    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" containerName="payload-processing"
Nov 03 01:26:51.445245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.443319    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:26:51.445245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.443444    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.448234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.448208    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b6afd6\""
Nov 03 01:26:51.448613 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.448595    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b6afd6-dockercfg-bprnb\""
Nov 03 01:26:51.511392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511355    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.511572 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xg6\" (UniqueName: \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.511687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511666    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:51.511726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511691    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28t4\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-kube-api-access-f28t4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:51.511726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511702    4531 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-routing-overlay-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:51.511726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.511711    4531 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687-provider-credentials\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:51.613113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.613019    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.613113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.613090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xg6\" (UniqueName: \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.613833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.613808    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.621696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.621671    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xg6\" (UniqueName: \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\") pod \"payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.673417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.673366    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" podUID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" containerName="payload-processing" containerID="cri-o://3c699ead40280b2082c951483ef7e22bf396b18a385b936ac627d76a2facb207" gracePeriod=25
Nov 03 01:26:51.765634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.765594    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:51.961201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.958612    4531 generic.go:356] "Generic (PLEG): container finished" podID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" containerID="3c699ead40280b2082c951483ef7e22bf396b18a385b936ac627d76a2facb207" exitCode=0
Nov 03 01:26:51.961201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.958806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" event={"ID":"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed","Type":"ContainerDied","Data":"3c699ead40280b2082c951483ef7e22bf396b18a385b936ac627d76a2facb207"}
Nov 03 01:26:51.966256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.966213    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" containerID="af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204" exitCode=0
Nov 03 01:26:51.966507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.966475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" event={"ID":"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687","Type":"ContainerDied","Data":"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"}
Nov 03 01:26:51.966587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.966550    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx" event={"ID":"2f7f9f60-3ee0-4e6a-ab1f-f273969b9687","Type":"ContainerDied","Data":"0e940f2c71b9ea357a81414112abe3cf9614385f998d194b2d2e8387d871529f"}
Nov 03 01:26:51.966641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.966590    4531 scope.go:122] "RemoveContainer" containerID="af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"
Nov 03 01:26:51.966827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.966808    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"
Nov 03 01:26:51.992755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.992732    4531 scope.go:122] "RemoveContainer" containerID="af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"
Nov 03 01:26:51.993147 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:26:51.993126    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204\": container with ID starting with af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204 not found: ID does not exist" containerID="af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"
Nov 03 01:26:51.993313 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:51.993155    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204"} err="failed to get container status \"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204\": rpc error: code = NotFound desc = could not find container \"af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204\": container with ID starting with af087cd7c9eb9819c4c5ef456ca9386459b1576be92f4d82d846e4a1b2dd7204 not found: ID does not exist"
Nov 03 01:26:52.005733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.005691    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:26:52.008828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.008804    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:26:52.013965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.013940    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-7c759a/payload-processing-external-model-e2e-extmod-a-7c759a-bd9c9xtwx"]
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.024331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\") pod \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") "
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.024387    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\") pod \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") "
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.024641    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\") pod \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") "
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.024738    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\") pod \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\" (UID: \"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed\") "
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.025522    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume" pod "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" (UID: "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.025790    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume" pod "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" (UID: "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.028116    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89" pod "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" (UID: "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed"). InnerVolumeSpecName "kube-api-access-h9h89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:52.031999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.030045    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials" pod "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" (UID: "93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:52.066248 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:26:52.066211    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b24bea7_3fe6_423f_88cf_7673f98e0b54.slice/crio-33a4adc6763f3e8db90b1b0c7d45ff6ccda5eabe56127a9b087794826a55edaa WatchSource:0}: Error finding container 33a4adc6763f3e8db90b1b0c7d45ff6ccda5eabe56127a9b087794826a55edaa: Status 404 returned error can't find the container with id 33a4adc6763f3e8db90b1b0c7d45ff6ccda5eabe56127a9b087794826a55edaa
Nov 03 01:26:52.067479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.067452    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:26:52.127429 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.127405    4531 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-provider-credentials\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:52.127528 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.127434    4531 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-routing-overlay-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:52.127528 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.127451    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:52.127528 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.127464    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h89\" (UniqueName: \"kubernetes.io/projected/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed-kube-api-access-h9h89\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:26:52.906570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.906532    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7f9f60-3ee0-4e6a-ab1f-f273969b9687" path="/var/lib/kubelet/pods/2f7f9f60-3ee0-4e6a-ab1f-f273969b9687/volumes"
Nov 03 01:26:52.983943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.983911    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"
Nov 03 01:26:52.984148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.983946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4" event={"ID":"93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed","Type":"ContainerDied","Data":"c70481cd00a6d2a670bf131edfa935b18f87586aba6f205176ffbd866f88d8b7"}
Nov 03 01:26:52.984148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.984019    4531 scope.go:122] "RemoveContainer" containerID="3c699ead40280b2082c951483ef7e22bf396b18a385b936ac627d76a2facb207"
Nov 03 01:26:52.986020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.985991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" event={"ID":"8b24bea7-3fe6-423f-88cf-7673f98e0b54","Type":"ContainerStarted","Data":"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"}
Nov 03 01:26:52.986145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.986045    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" event={"ID":"8b24bea7-3fe6-423f-88cf-7673f98e0b54","Type":"ContainerStarted","Data":"33a4adc6763f3e8db90b1b0c7d45ff6ccda5eabe56127a9b087794826a55edaa"}
Nov 03 01:26:52.986145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:52.986124    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:26:53.030466 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:53.030400    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" podStartSLOduration=40.030380225 podStartE2EDuration="40.030380225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:53.009402515 +0000 UTC m=+2758.223146188" watchObservedRunningTime="2026-11-03 01:26:53.030380225 +0000 UTC m=+2758.244123898"
Nov 03 01:26:53.061576 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:53.061531    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:26:53.067410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:53.067374    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-57079b/payload-processing-external-model-e2e-extmod-b-57079b-6b9d5h6l4"]
Nov 03 01:26:54.936080 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:26:54.932576    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" path="/var/lib/kubelet/pods/93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed/volumes"
Nov 03 01:27:03.999526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:03.999493    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:27:09.000615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.000580    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:27:09.001632 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.001595    4531 cpu_manager.go:414] "containerMap: removing container" podUID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" containerName="payload-processing"
Nov 03 01:27:09.001632 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.001618    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" containerName="payload-processing"
Nov 03 01:27:09.001827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.001794    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fb8a74-6f52-4a3f-af91-9afd7f8ab1ed" containerName="payload-processing"
Nov 03 01:27:09.008346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.008292    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.012345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.012317    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:27:09.015831 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.015805    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5bfc51\""
Nov 03 01:27:09.036709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.036672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tcf\" (UniqueName: \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.038066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.038040    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.142789 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.142378    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.142789 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.142574    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tcf\" (UniqueName: \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.146183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.144098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.195953 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.195914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tcf\" (UniqueName: \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\") pod \"payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.356207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.353956    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:09.686457 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:09.686428    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:27:09.686811 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:27:09.686785    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc1cf63_4645_4a69_ae98_5fa2dafa7fbd.slice/crio-f7170db3741ceff838e595efd45af41b2c0f0ca4b3efdef5d9d6eb94ad885dd2 WatchSource:0}: Error finding container f7170db3741ceff838e595efd45af41b2c0f0ca4b3efdef5d9d6eb94ad885dd2: Status 404 returned error can't find the container with id f7170db3741ceff838e595efd45af41b2c0f0ca4b3efdef5d9d6eb94ad885dd2
Nov 03 01:27:09.880288 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:27:09.880251    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:27:10.235095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:10.235052    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" event={"ID":"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd","Type":"ContainerStarted","Data":"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"}
Nov 03 01:27:10.235652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:10.235120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" event={"ID":"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd","Type":"ContainerStarted","Data":"f7170db3741ceff838e595efd45af41b2c0f0ca4b3efdef5d9d6eb94ad885dd2"}
Nov 03 01:27:10.235652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:10.235277    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:10.254231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:10.254154    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" podStartSLOduration=2.254136964 podStartE2EDuration="2.254136964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:27:10.252199114 +0000 UTC m=+2775.465942787" watchObservedRunningTime="2026-11-03 01:27:10.254136964 +0000 UTC m=+2775.467880637"
Nov 03 01:27:21.251344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:27:21.251307    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:27:39.880081 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:27:39.880042    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:28:09.881701 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:28:09.881615    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:28:22.638333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:22.638287    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:28:22.641449 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:22.638649    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" podUID="7094882d-8427-4d75-ac29-73e788cd33c0" containerName="maas-api" containerID="cri-o://84902bef4eaeab3451b9c8ae153284123f233367dd4b7bc188fc5f6f9719b323" gracePeriod=30
Nov 03 01:28:23.304246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.304045    4531 generic.go:356] "Generic (PLEG): container finished" podID="7094882d-8427-4d75-ac29-73e788cd33c0" containerID="84902bef4eaeab3451b9c8ae153284123f233367dd4b7bc188fc5f6f9719b323" exitCode=0
Nov 03 01:28:23.310291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.304293    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" event={"ID":"7094882d-8427-4d75-ac29-73e788cd33c0","Type":"ContainerDied","Data":"84902bef4eaeab3451b9c8ae153284123f233367dd4b7bc188fc5f6f9719b323"}
Nov 03 01:28:23.352126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.352097    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:28:23.390796 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.390751    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:28:23.392822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.392181    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7094882d-8427-4d75-ac29-73e788cd33c0" containerName="maas-api"
Nov 03 01:28:23.392822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.392208    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7094882d-8427-4d75-ac29-73e788cd33c0" containerName="maas-api"
Nov 03 01:28:23.392822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.392428    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7094882d-8427-4d75-ac29-73e788cd33c0" containerName="maas-api"
Nov 03 01:28:23.402065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.402032    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:28:23.403095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.403069    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.407955 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.407936    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.408525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.408505    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:28:23.414850 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.414660    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6fec8a-dockercfg-nlkgj\""
Nov 03 01:28:23.414850 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.414812    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6fec8a\""
Nov 03 01:28:23.423622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.423595    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:28:23.448529 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.448496    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\") pod \"7094882d-8427-4d75-ac29-73e788cd33c0\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") "
Nov 03 01:28:23.448669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.448597    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\") pod \"7094882d-8427-4d75-ac29-73e788cd33c0\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") "
Nov 03 01:28:23.448958 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.448937    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.449068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.449052    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hhb\" (UniqueName: \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.449123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.449108    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9svc\" (UniqueName: \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.449211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.449173    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.449596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.449574    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca" pod "7094882d-8427-4d75-ac29-73e788cd33c0" (UID: "7094882d-8427-4d75-ac29-73e788cd33c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:23.455704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.455675    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls" pod "7094882d-8427-4d75-ac29-73e788cd33c0" (UID: "7094882d-8427-4d75-ac29-73e788cd33c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:23.552496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.552452    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\") pod \"7094882d-8427-4d75-ac29-73e788cd33c0\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") "
Nov 03 01:28:23.552663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.552569    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\" (UniqueName: \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\") pod \"7094882d-8427-4d75-ac29-73e788cd33c0\" (UID: \"7094882d-8427-4d75-ac29-73e788cd33c0\") "
Nov 03 01:28:23.552932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.552902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hhb\" (UniqueName: \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.553104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.552972    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9svc\" (UniqueName: \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.553104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.553025    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.553357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.553337    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.553490 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.553455    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:23.553490 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.553474    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7094882d-8427-4d75-ac29-73e788cd33c0-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:23.555623 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.555565    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.557574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.557550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.560282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.560237    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs" pod "7094882d-8427-4d75-ac29-73e788cd33c0" (UID: "7094882d-8427-4d75-ac29-73e788cd33c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:23.560576 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.560551    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f" pod "7094882d-8427-4d75-ac29-73e788cd33c0" (UID: "7094882d-8427-4d75-ac29-73e788cd33c0"). InnerVolumeSpecName "kube-api-access-96x6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:23.564732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.564708    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9svc\" (UniqueName: \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\") pod \"payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:23.564732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.564724    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hhb\" (UniqueName: \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\") pod \"payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.687184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.686542    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7094882d-8427-4d75-ac29-73e788cd33c0-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:23.687184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.686588    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96x6f\" (UniqueName: \"kubernetes.io/projected/7094882d-8427-4d75-ac29-73e788cd33c0-kube-api-access-96x6f\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:23.733484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.733447    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:23.758246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:23.757984    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:24.234983 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.234905    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:28:24.239118 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:28:24.239086    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e2f601_0529_4a46_8566_2fce413f8879.slice/crio-98f827daff4659f391825adace0715bfbda6dd4fda29fce34692842556015b6c WatchSource:0}: Error finding container 98f827daff4659f391825adace0715bfbda6dd4fda29fce34692842556015b6c: Status 404 returned error can't find the container with id 98f827daff4659f391825adace0715bfbda6dd4fda29fce34692842556015b6c
Nov 03 01:28:24.268925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.247895    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:28:24.328255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.328124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" event={"ID":"35e2f601-0529-4a46-8566-2fce413f8879","Type":"ContainerStarted","Data":"98f827daff4659f391825adace0715bfbda6dd4fda29fce34692842556015b6c"}
Nov 03 01:28:24.330764 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.330730    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc" event={"ID":"7094882d-8427-4d75-ac29-73e788cd33c0","Type":"ContainerDied","Data":"1578028b7f387af7b9f061fec35477725d18296a3f6edb44bfe3f5e06d36fff4"}
Nov 03 01:28:24.330889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.330797    4531 scope.go:122] "RemoveContainer" containerID="84902bef4eaeab3451b9c8ae153284123f233367dd4b7bc188fc5f6f9719b323"
Nov 03 01:28:24.331216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.330981    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"
Nov 03 01:28:24.338289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.338218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" event={"ID":"7f72c0ac-adef-4be1-bac6-3078aebbb44d","Type":"ContainerStarted","Data":"1d43635b53d91537c66ec689134633c3334d39a90d1c978da70f4e0213be5f27"}
Nov 03 01:28:24.390281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.376195    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:28:24.390281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.384464    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a88862-59974c8bc7-lwqcc"]
Nov 03 01:28:24.907588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:24.907554    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7094882d-8427-4d75-ac29-73e788cd33c0" path="/var/lib/kubelet/pods/7094882d-8427-4d75-ac29-73e788cd33c0/volumes"
Nov 03 01:28:25.351949 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.351902    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" event={"ID":"7f72c0ac-adef-4be1-bac6-3078aebbb44d","Type":"ContainerStarted","Data":"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"}
Nov 03 01:28:25.352245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.352209    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:25.353790 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.353763    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" event={"ID":"35e2f601-0529-4a46-8566-2fce413f8879","Type":"ContainerStarted","Data":"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"}
Nov 03 01:28:25.353922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.353823    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:25.379558 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.379123    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" podStartSLOduration=17.379105288 podStartE2EDuration="17.379105288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:25.372640181 +0000 UTC m=+2850.586383855" watchObservedRunningTime="2026-11-03 01:28:25.379105288 +0000 UTC m=+2850.592848964"
Nov 03 01:28:25.390973 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:25.390926    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" podStartSLOduration=17.390908385 podStartE2EDuration="17.390908385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:25.388020366 +0000 UTC m=+2850.601764040" watchObservedRunningTime="2026-11-03 01:28:25.390908385 +0000 UTC m=+2850.604652057"
Nov 03 01:28:33.576560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:33.576517    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:28:33.577381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:33.577348    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" podUID="716fa21a-6f33-4ee3-99d1-a24e2062d671" containerName="maas-api" containerID="cri-o://a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19" gracePeriod=30
Nov 03 01:28:34.322382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.322357    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:28:34.333063 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333028    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\") pod \"716fa21a-6f33-4ee3-99d1-a24e2062d671\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") "
Nov 03 01:28:34.333272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333077    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\") pod \"716fa21a-6f33-4ee3-99d1-a24e2062d671\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") "
Nov 03 01:28:34.333272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333209    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\") pod \"716fa21a-6f33-4ee3-99d1-a24e2062d671\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") "
Nov 03 01:28:34.333272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333245    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\" (UniqueName: \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\") pod \"716fa21a-6f33-4ee3-99d1-a24e2062d671\" (UID: \"716fa21a-6f33-4ee3-99d1-a24e2062d671\") "
Nov 03 01:28:34.333465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333419    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca" pod "716fa21a-6f33-4ee3-99d1-a24e2062d671" (UID: "716fa21a-6f33-4ee3-99d1-a24e2062d671"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:34.333982 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.333955    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716fa21a-6f33-4ee3-99d1-a24e2062d671-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:34.335767 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.335738    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c" pod "716fa21a-6f33-4ee3-99d1-a24e2062d671" (UID: "716fa21a-6f33-4ee3-99d1-a24e2062d671"). InnerVolumeSpecName "kube-api-access-7k54c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:34.335767 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.335755    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs" pod "716fa21a-6f33-4ee3-99d1-a24e2062d671" (UID: "716fa21a-6f33-4ee3-99d1-a24e2062d671"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:34.335959 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.335781    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls" pod "716fa21a-6f33-4ee3-99d1-a24e2062d671" (UID: "716fa21a-6f33-4ee3-99d1-a24e2062d671"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:28:34.358575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.357151    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:28:34.358575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.358453    4531 cpu_manager.go:414] "containerMap: removing container" podUID="716fa21a-6f33-4ee3-99d1-a24e2062d671" containerName="maas-api"
Nov 03 01:28:34.358575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.358476    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716fa21a-6f33-4ee3-99d1-a24e2062d671" containerName="maas-api"
Nov 03 01:28:34.358859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.358677    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="716fa21a-6f33-4ee3-99d1-a24e2062d671" containerName="maas-api"
Nov 03 01:28:34.363255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.363228    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.373327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.369948    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee618c-openshift-default-dockercfg-d6ps5\""
Nov 03 01:28:34.378335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.377993    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:28:34.435001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.434905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435016    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435313    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435445 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435412    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zrw\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435476    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435538    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.435706 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435690    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:34.435804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435710    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k54c\" (UniqueName: \"kubernetes.io/projected/716fa21a-6f33-4ee3-99d1-a24e2062d671-kube-api-access-7k54c\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:34.435804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.435729    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716fa21a-6f33-4ee3-99d1-a24e2062d671-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:34.441696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.441664    4531 generic.go:356] "Generic (PLEG): container finished" podID="716fa21a-6f33-4ee3-99d1-a24e2062d671" containerID="a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19" exitCode=0
Nov 03 01:28:34.441823 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.441746    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"
Nov 03 01:28:34.441823 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.441752    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" event={"ID":"716fa21a-6f33-4ee3-99d1-a24e2062d671","Type":"ContainerDied","Data":"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"}
Nov 03 01:28:34.441913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.441850    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm" event={"ID":"716fa21a-6f33-4ee3-99d1-a24e2062d671","Type":"ContainerDied","Data":"55859b3a9e3093d16af2453dbb5c2fd24cb5e765b14f60db3688b2a2c3a23192"}
Nov 03 01:28:34.441913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.441879    4531 scope.go:122] "RemoveContainer" containerID="a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"
Nov 03 01:28:34.463567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.463536    4531 scope.go:122] "RemoveContainer" containerID="a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"
Nov 03 01:28:34.498194 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:28:34.496325    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19\": container with ID starting with a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19 not found: ID does not exist" containerID="a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"
Nov 03 01:28:34.498194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.496383    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19"} err="failed to get container status \"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19\": rpc error: code = NotFound desc = could not find container \"a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19\": container with ID starting with a94e7e583dcaf450bcdde8c8197a64d1f89d1339045c776c7af73423fd95bf19 not found: ID does not exist"
Nov 03 01:28:34.522282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.522230    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:28:34.530443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.530402    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce04af-554767746d-z5fdm"]
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.536992    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537082    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537126    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537177    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537232    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zrw\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537405    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537447    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.537803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.537637    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.538483 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.538281    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.538543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.538513    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.541904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.541873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.542415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.542388    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.544311 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.544287    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.549660 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.549630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.554360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.554323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zrw\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.554496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.554360    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.720875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.720835    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:34.906141 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:34.906101    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716fa21a-6f33-4ee3-99d1-a24e2062d671" path="/var/lib/kubelet/pods/716fa21a-6f33-4ee3-99d1-a24e2062d671/volumes"
Nov 03 01:28:35.023083 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:28:35.023051    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6018817e_d13d_42cd_b420_3a15c387c78a.slice/crio-cf10e36fe47497ab0691f91ea8b1dd5a618e011ae6157328395d55dfdf959696 WatchSource:0}: Error finding container cf10e36fe47497ab0691f91ea8b1dd5a618e011ae6157328395d55dfdf959696: Status 404 returned error can't find the container with id cf10e36fe47497ab0691f91ea8b1dd5a618e011ae6157328395d55dfdf959696
Nov 03 01:28:35.023616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.023596    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:28:35.026047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.026007    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:35.026132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.026085    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:35.026132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.026114    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:35.454889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.454847    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" event={"ID":"6018817e-d13d-42cd-b420-3a15c387c78a","Type":"ContainerStarted","Data":"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"}
Nov 03 01:28:35.455113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.454913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" event={"ID":"6018817e-d13d-42cd-b420-3a15c387c78a","Type":"ContainerStarted","Data":"cf10e36fe47497ab0691f91ea8b1dd5a618e011ae6157328395d55dfdf959696"}
Nov 03 01:28:35.480856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.480789    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podStartSLOduration=91.480768675 podStartE2EDuration="1m31.480768675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:35.473708185 +0000 UTC m=+2860.687451881" watchObservedRunningTime="2026-11-03 01:28:35.480768675 +0000 UTC m=+2860.694512348"
Nov 03 01:28:35.723279 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:35.721255    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:36.394516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:36.393119    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:28:36.410769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:36.410344    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:28:36.723973 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:36.723894    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:36.723973 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:36.723948    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:37.721618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:37.721578    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:37.721779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:37.721636    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:38.721663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:38.721623    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:38.722301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:38.722262    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:39.721786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:39.721749    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:39.722256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:39.721806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:39.890036 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:28:39.889895    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:28:40.225670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:40.225628    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:28:40.226130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:40.226098    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy" containerID="cri-o://d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63" gracePeriod=30
Nov 03 01:28:40.721594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:40.721555    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:40.721792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:40.721616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:41.742864 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:41.742827    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 01:28:41.743498 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:41.743153    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 01:28:42.727443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:42.727407    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:42.727905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:42.727885    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:42.728819 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:42.728795    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:28:45.557450 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.557420    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:28:45.571345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571314    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571536 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571360    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571536 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571467    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571536 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571523    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571661    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:28:45.571873 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571784    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571873 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571819    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.571978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.571882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\") pod \"ac858873-189e-4f6a-b104-3c698fe7c16a\" (UID: \"ac858873-189e-4f6a-b104-3c698fe7c16a\") "
Nov 03 01:28:45.572370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.572239    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:28:45.572959 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.572741    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:28:45.573278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.573198    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.573278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.573232    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.573278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.573249    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.573278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.573201    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:45.573614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.573585    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:28:45.574695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.574670    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:28:45.576920 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.576884    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:28:45.577006 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.576923    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "kube-api-access-d4vbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:45.584987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.584941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token" pod "ac858873-189e-4f6a-b104-3c698fe7c16a" (UID: "ac858873-189e-4f6a-b104-3c698fe7c16a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:45.596111 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.596077    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:28:45.597458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.597333    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy"
Nov 03 01:28:45.597458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.597357    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy"
Nov 03 01:28:45.597642 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.597599    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerName="istio-proxy"
Nov 03 01:28:45.605433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.604118    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.609869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.609844    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:28:45.674599 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674552    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.674773 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.674866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674771    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.674866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674832    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.674866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674857    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.675034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.674972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.675106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675048    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxv5\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.675106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.675251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675232    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.675386 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675371    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac858873-189e-4f6a-b104-3c698fe7c16a-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.675448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675392    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.675448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675407    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.675448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675422    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.675448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675436    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vbl\" (UniqueName: \"kubernetes.io/projected/ac858873-189e-4f6a-b104-3c698fe7c16a-kube-api-access-d4vbl\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.675666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.675451    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac858873-189e-4f6a-b104-3c698fe7c16a-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:28:45.762535 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.762488    4531 generic.go:356] "Generic (PLEG): container finished" podID="ac858873-189e-4f6a-b104-3c698fe7c16a" containerID="d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63" exitCode=0
Nov 03 01:28:45.762731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.762664    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" event={"ID":"ac858873-189e-4f6a-b104-3c698fe7c16a","Type":"ContainerDied","Data":"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"}
Nov 03 01:28:45.762731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.762717    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h" event={"ID":"ac858873-189e-4f6a-b104-3c698fe7c16a","Type":"ContainerDied","Data":"79a3dab8f9b1fbf4bb389760bc83cacdbffa207031789d1c16b6106d78a1afde"}
Nov 03 01:28:45.762863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.762747    4531 scope.go:122] "RemoveContainer" containerID="d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"
Nov 03 01:28:45.762943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.762925    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.780297    4531 scope.go:122] "RemoveContainer" containerID="d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.781451    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxv5\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.781704    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.781876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.781952    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782029    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782079    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782143    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782323    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.782773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.783030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.783295    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.783998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.783546    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.785104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.784243    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.785814 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.785789    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.786554 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.786529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.787284 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:28:45.787244    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63\": container with ID starting with d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63 not found: ID does not exist" containerID="d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"
Nov 03 01:28:45.787407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.787282    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63"} err="failed to get container status \"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63\": rpc error: code = NotFound desc = could not find container \"d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63\": container with ID starting with d5f8d076db2a02c19d501e49d046c56b76c0d441b4ed8cb983847658f8194a63 not found: ID does not exist"
Nov 03 01:28:45.790358 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.790335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxv5\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.790589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.790524    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\") pod \"e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") " pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:45.800908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.800875    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:28:45.810451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.810415    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce04af-openshift-default-7dccd45f47-p548h"]
Nov 03 01:28:45.950189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:45.943545    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:46.279148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.279116    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:28:46.283537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.283497    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:46.283662 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.283582    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:46.283662 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.283622    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:28:46.822197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.818809    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" event={"ID":"429ff11e-60e8-4d41-a6e3-c2981e342b1b","Type":"ContainerStarted","Data":"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"}
Nov 03 01:28:46.822197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.818919    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" event={"ID":"429ff11e-60e8-4d41-a6e3-c2981e342b1b","Type":"ContainerStarted","Data":"e46d1174c8fe04615cc131dd3437b14c4a0cd7de28b799b467b960c201575b44"}
Nov 03 01:28:46.847056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.847004    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podStartSLOduration=102.846988421 podStartE2EDuration="1m42.846988421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:46.841454035 +0000 UTC m=+2872.055197717" watchObservedRunningTime="2026-11-03 01:28:46.846988421 +0000 UTC m=+2872.060732093"
Nov 03 01:28:46.908080 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.908047    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac858873-189e-4f6a-b104-3c698fe7c16a" path="/var/lib/kubelet/pods/ac858873-189e-4f6a-b104-3c698fe7c16a/volumes"
Nov 03 01:28:46.955324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:46.955277    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:47.958331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:47.958235    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:28:47.958331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:47.958303    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:28:48.958499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:48.958460    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:28:48.959019 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:48.958541    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:28:49.955472 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:49.955432    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:28:49.955754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:49.955721    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:28:50.957401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:50.957362    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:28:50.958000 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:50.957969    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:28:51.957748 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:51.957707    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:28:51.958416 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:51.958358    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:28:52.960282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:52.960250    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:52.960739 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:52.960688    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:28:52.961640 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:28:52.961612    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:29:09.879831 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:29:09.879793    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:29:37.495228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:37.495148    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" podUID="df529571-d827-4799-9b09-3e0ec8b4ce56" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.37:8081/readyz\": dial tcp 10.133.0.37:8081: connect: connection refused"
Nov 03 01:29:37.676101 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:37.676070    4531 generic.go:356] "Generic (PLEG): container finished" podID="df529571-d827-4799-9b09-3e0ec8b4ce56" containerID="c48db5dca8a3ed3891e64f2d686edef2508c777256b0d86e0c9ebdda7dd21a1e" exitCode=1
Nov 03 01:29:37.676316 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:37.676152    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" event={"ID":"df529571-d827-4799-9b09-3e0ec8b4ce56","Type":"ContainerDied","Data":"c48db5dca8a3ed3891e64f2d686edef2508c777256b0d86e0c9ebdda7dd21a1e"}
Nov 03 01:29:37.676657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:37.676642    4531 scope.go:122] "RemoveContainer" containerID="c48db5dca8a3ed3891e64f2d686edef2508c777256b0d86e0c9ebdda7dd21a1e"
Nov 03 01:29:37.680996 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:37.680977    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 01:29:38.687811 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:38.687771    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f" event={"ID":"df529571-d827-4799-9b09-3e0ec8b4ce56","Type":"ContainerStarted","Data":"94f7dfd4de5d8ea7575e27356568cb49792fc8ea49ced8ec595ce75151c85ffb"}
Nov 03 01:29:38.688331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:38.688018    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 01:29:39.881180 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:29:39.881130    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:29:49.699270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:29:49.699233    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-sw79f"
Nov 03 01:30:00.178240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.178196    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:00.188044 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.184353    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.188044 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.186039    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:00.193222 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.193193    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.194946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.194925    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:00.200359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.200338    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.201328 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.201308    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:00.207395 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.207374    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.210025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.210006    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:00.219153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.219133    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:00.226796 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.226774    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:00.256421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.256384    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6q4xp\""
Nov 03 01:30:00.262700 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.262670    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:00.339940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.339895    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfz4\" (UniqueName: \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.340183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340032    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.340183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340078    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.340296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340273    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj8s\" (UniqueName: \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.340340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340328    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89ksw\" (UniqueName: \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.340415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340392    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.340468 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340455    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpc7p\" (UniqueName: \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.340630 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.340617    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.441994 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.441889    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.442205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.441998    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfz4\" (UniqueName: \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.442205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442054    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.442345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.442730 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442674    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.442881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442851    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.442993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442896    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.442993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj8s\" (UniqueName: \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.442993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442977    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ksw\" (UniqueName: \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.443198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.442998    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.443198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.443039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpc7p\" (UniqueName: \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.443641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.443619    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.450300 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.450275    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfz4\" (UniqueName: \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\") pod \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.451527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.451504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ksw\" (UniqueName: \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\") pod \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.451618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.451556    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpc7p\" (UniqueName: \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\") pod \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.452021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.452003    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj8s\" (UniqueName: \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:00.522343 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.522301    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:00.527967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.527932    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:00.536077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.536052    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:00.554905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:00.554662    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:02.254570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:02.254507    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:30:02.255077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:02.254779    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy" containerID="cri-o://493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9" gracePeriod=30
Nov 03 01:30:03.135289 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.135241    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:03.135289 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60" Netns:"/var/run/netns/586580ec-e1b4-4f28-b493-f2b673428b4b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.135289 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.135289 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.135321    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60" Netns:"/var/run/netns/586580ec-e1b4-4f28-b493-f2b673428b4b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.135346    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60" Netns:"/var/run/netns/586580ec-e1b4-4f28-b493-f2b673428b4b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:03.135592 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.135438    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60\\\" Netns:\\\"/var/run/netns/586580ec-e1b4-4f28-b493-f2b673428b4b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=4bc579fea5fcd5219d5e1a5c795e307f51a5c82fce38f1a35f91f7d3bb034d60;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941"
Nov 03 01:30:03.161943 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.161897    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:03.161943 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104" Netns:"/var/run/netns/5d55b183-cb0e-4678-bfc5-7528b3ecb6cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.161943 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.161943 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.161966    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104" Netns:"/var/run/netns/5d55b183-cb0e-4678-bfc5-7528b3ecb6cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.161989    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104" Netns:"/var/run/netns/5d55b183-cb0e-4678-bfc5-7528b3ecb6cf" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:03.162274 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.162082    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104\\\" Netns:\\\"/var/run/netns/5d55b183-cb0e-4678-bfc5-7528b3ecb6cf\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=31c30f5575dee20c7474fb7b58a8945798839cfe13dede475e9228716a44b104;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11"
Nov 03 01:30:03.178794 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178751    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:03.178794 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92" Netns:"/var/run/netns/f1faa35f-c928-4666-ba82-899b77c6da3d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.178794 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.178794 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178785    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6" Netns:"/var/run/netns/3ae625d1-22ce-4cab-b31a-d61469351790" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178815    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92" Netns:"/var/run/netns/f1faa35f-c928-4666-ba82-899b77c6da3d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178831    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6" Netns:"/var/run/netns/3ae625d1-22ce-4cab-b31a-d61469351790" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178840    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92" Netns:"/var/run/netns/f1faa35f-c928-4666-ba82-899b77c6da3d" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178852    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6" Netns:"/var/run/netns/3ae625d1-22ce-4cab-b31a-d61469351790" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178931    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92\\\" Netns:\\\"/var/run/netns/f1faa35f-c928-4666-ba82-899b77c6da3d\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=9e0d6d16316eaf559bd77f062cc6a1e9cf402bba4640e74fd47089f08020fa92;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7"
Nov 03 01:30:03.179072 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:03.178931    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6\\\" Netns:\\\"/var/run/netns/3ae625d1-22ce-4cab-b31a-d61469351790\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=9eaf9048b31e2fefb330624274140a9ee1b08cb93477edb69ac4bebfdb64dff6;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2"
Nov 03 01:30:04.005294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.005259    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:04.005294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.005297    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.005270    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.005523    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.005818    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.006066    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.006148    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:04.006960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:04.006228    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:05.981200 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:05.979063    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 01:30:06.064231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:06.061725    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy" containerID="cri-o://153f11294e8bfbd5afea776f4c13ce8c9835d136416387803f61596d25e9ff77" gracePeriod=30
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.719555    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1" Netns:"/var/run/netns/14f89b73-60c6-4f64-b843-8f95d1aea405" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.719610    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1" Netns:"/var/run/netns/14f89b73-60c6-4f64-b843-8f95d1aea405" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.719628    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1" Netns:"/var/run/netns/14f89b73-60c6-4f64-b843-8f95d1aea405" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:06.719967 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.719702    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra_b807a829-b4a3-4df2-ac05-9bf0601b4e11_0(2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1\\\" Netns:\\\"/var/run/netns/14f89b73-60c6-4f64-b843-8f95d1aea405\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm;K8S_POD_INFRA_CONTAINER_ID=2003ee6352cf6b7a7efc00f51966c70dc4e88e82f4313a839e59e7214466fef1;K8S_POD_UID=b807a829-b4a3-4df2-ac05-9bf0601b4e11\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm/b807a829-b4a3-4df2-ac05-9bf0601b4e11]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11"
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.820412    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d" Netns:"/var/run/netns/c7ce26c3-f087-44ec-9919-03326cee3305" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.820523    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d" Netns:"/var/run/netns/c7ce26c3-f087-44ec-9919-03326cee3305" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.820528 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:06.820947 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.820547    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:06.820947 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d" Netns:"/var/run/netns/c7ce26c3-f087-44ec-9919-03326cee3305" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s": context deadline exceeded
Nov 03 01:30:06.820947 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:06.820947 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:06.820947 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:06.820674    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra_76f53f02-45ad-44bd-bf1e-f519f5332df2_0(20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d\\\" Netns:\\\"/var/run/netns/c7ce26c3-f087-44ec-9919-03326cee3305\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9;K8S_POD_INFRA_CONTAINER_ID=20ca5f73489cce9e6fa03b64985bb193eaa7da878887cdac4f9c437bf953798d;K8S_POD_UID=76f53f02-45ad-44bd-bf1e-f519f5332df2\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9/76f53f02-45ad-44bd-bf1e-f519f5332df2]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2"
Nov 03 01:30:07.030899 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.030847    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:07.030899 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c" Netns:"/var/run/netns/f7cded18-e826-44b5-86e2-9d6b9e6c5ddd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.030899 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.030899 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:07.055321 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.055271    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:07.055321 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c" Netns:"/var/run/netns/f7cded18-e826-44b5-86e2-9d6b9e6c5ddd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.055321 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.055321 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:07.062263 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.062208    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:07.062263 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c" Netns:"/var/run/netns/f7cded18-e826-44b5-86e2-9d6b9e6c5ddd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.062263 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.062263 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:07.062791 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.062727    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra_64550299-3851-43d1-b2ac-4a2ddb450ac7_0(a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6 to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c\\\" Netns:\\\"/var/run/netns/f7cded18-e826-44b5-86e2-9d6b9e6c5ddd\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6;K8S_POD_INFRA_CONTAINER_ID=a933ccbb3aecff61d4257c92331cd9f4c7270906081373cfc918e1dc1b72583c;K8S_POD_UID=64550299-3851-43d1-b2ac-4a2ddb450ac7\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6/64550299-3851-43d1-b2ac-4a2ddb450ac7]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7"
Nov 03 01:30:07.100180 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.100117    4531 log.go:32] "RunPodSandbox from runtime service failed" err=<
Nov 03 01:30:07.100180 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7" Netns:"/var/run/netns/83b79d0b-20cd-40d6-841f-c56c5fcc355b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.100180 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.100180 ip-10-0-136-112 kubenswrapper[4531]:  >
Nov 03 01:30:07.100582 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.100558    4531 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Nov 03 01:30:07.100582 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7" Netns:"/var/run/netns/83b79d0b-20cd-40d6-841f-c56c5fcc355b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.100582 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.100582 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:07.100850 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.100826    4531 kuberuntime_manager.go:1594] "CreatePodSandbox for pod failed" err=<
Nov 03 01:30:07.100850 ip-10-0-136-112 kubenswrapper[4531]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7" Netns:"/var/run/netns/83b79d0b-20cd-40d6-841f-c56c5fcc355b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941" Path:"" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get "https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s": context deadline exceeded
Nov 03 01:30:07.100850 ip-10-0-136-112 kubenswrapper[4531]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Nov 03 01:30:07.100850 ip-10-0-136-112 kubenswrapper[4531]:  > pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:07.101216 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:07.101143    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra_dc1063ff-cc0e-43fd-ace6-348f52a17941_0(c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7): error adding pod odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7\\\" Netns:\\\"/var/run/netns/83b79d0b-20cd-40d6-841f-c56c5fcc355b\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=odh-ai-gateway-infra;K8S_POD_NAME=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv;K8S_POD_INFRA_CONTAINER_ID=c961c91536ca32ed3feb43a2c7db641f0957e1eac3471cc18df26c71e495a7e7;K8S_POD_UID=dc1063ff-cc0e-43fd-ace6-348f52a17941\\\" Path:\\\"\\\" ERRORED: error configuring pod [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv] networking: Multus: [odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv/dc1063ff-cc0e-43fd-ace6-348f52a17941]: error waiting for pod: Get \\\"https://a5a811d24c7c444d185befcbe55e66ce-56158406077f4c60.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/pods/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941"
Nov 03 01:30:08.321406 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.321377    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:30:08.371238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.359601    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:30:08.371238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.361038    4531 cpu_manager.go:414] "containerMap: removing container" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy"
Nov 03 01:30:08.371238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.361063    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy"
Nov 03 01:30:08.371238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.361330    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" containerName="istio-proxy"
Nov 03 01:30:08.384271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.380642    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:30:08.384271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.380769    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.412966 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.412929    4531 generic.go:356] "Generic (PLEG): container finished" podID="be64d1aa-dda7-4951-988c-abdf2854a300" containerID="493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9" exitCode=0
Nov 03 01:30:08.413213 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.412977    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"
Nov 03 01:30:08.413312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.412993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" event={"ID":"be64d1aa-dda7-4951-988c-abdf2854a300","Type":"ContainerDied","Data":"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"}
Nov 03 01:30:08.413372 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.413336    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm" event={"ID":"be64d1aa-dda7-4951-988c-abdf2854a300","Type":"ContainerDied","Data":"24d00742c051adcc243d3c2e9c75b86505cd4245c70250a77b49059a483df5fa"}
Nov 03 01:30:08.413424 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.413372    4531 scope.go:122] "RemoveContainer" containerID="493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"
Nov 03 01:30:08.421324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421298    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.421567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421540    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:08.421627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421595    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.421693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421683    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.421749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421718    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.421749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421740    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.421858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.421789    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.422062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422039    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:08.422155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:08.422459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422440    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.422586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.422586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422537    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\") pod \"be64d1aa-dda7-4951-988c-abdf2854a300\" (UID: \"be64d1aa-dda7-4951-988c-abdf2854a300\") "
Nov 03 01:30:08.422985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.422957    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423036    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423187    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:08.423440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423307    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:08.423440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423328    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423406    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423455    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423496    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423573    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423629    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ms8\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423703    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be64d1aa-dda7-4951-988c-abdf2854a300-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423712    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423722    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423732    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.423834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.423741    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.425080 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.425047    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:08.425917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.425896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "kube-api-access-dp48f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:08.427587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.427287    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:08.427587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.427315    4531 scope.go:122] "RemoveContainer" containerID="493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"
Nov 03 01:30:08.427736 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:08.427643    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9\": container with ID starting with 493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9 not found: ID does not exist" containerID="493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"
Nov 03 01:30:08.427736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.427676    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9"} err="failed to get container status \"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9\": rpc error: code = NotFound desc = could not find container \"493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9\": container with ID starting with 493c1f2f3c5cc01985e5d6bb6960608d184cf9ddc22a2ac668bd5c6094349be9 not found: ID does not exist"
Nov 03 01:30:08.429555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.429523    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy" pod "be64d1aa-dda7-4951-988c-abdf2854a300" (UID: "be64d1aa-dda7-4951-988c-abdf2854a300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:08.524829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.524790    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.524871    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.524926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.524978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525032 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ms8\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525278    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525295    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525334    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525369    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525494    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.525663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525510    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp48f\" (UniqueName: \"kubernetes.io/projected/be64d1aa-dda7-4951-988c-abdf2854a300-kube-api-access-dp48f\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.525663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525527    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be64d1aa-dda7-4951-988c-abdf2854a300-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.525663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525543    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be64d1aa-dda7-4951-988c-abdf2854a300-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:08.525663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525578    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.525916 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.525894    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.526133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.526111    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.526329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.526305    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.528319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.528287    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.528462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.528441    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.533451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.533425    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.534256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.534231    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ms8\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\") pod \"maas-default-gateway-openshift-default-596dc9545f-lsz2g\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.713286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.711479    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:08.757947 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.757915    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:30:08.765502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.765468    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-4af1f7-openshift-default-745ff98565-kw7mm"]
Nov 03 01:30:08.910676 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:08.910638    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be64d1aa-dda7-4951-988c-abdf2854a300" path="/var/lib/kubelet/pods/be64d1aa-dda7-4951-988c-abdf2854a300/volumes"
Nov 03 01:30:09.883570 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:09.883520    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:30:09.893448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:09.893417    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:30:09.894480 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:09.894454    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f34cf8_614c_4856_8711_5f980c6a6ebe.slice/crio-110c56016716a8074bd49c241cb094558842b86f31aedbe21e73f7fe904067bd WatchSource:0}: Error finding container 110c56016716a8074bd49c241cb094558842b86f31aedbe21e73f7fe904067bd: Status 404 returned error can't find the container with id 110c56016716a8074bd49c241cb094558842b86f31aedbe21e73f7fe904067bd
Nov 03 01:30:09.897769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:09.897737    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:30:09.897895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:09.897807    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:30:09.897895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:09.897838    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:30:10.494829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.494778    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" event={"ID":"27f34cf8-614c-4856-8711-5f980c6a6ebe","Type":"ContainerStarted","Data":"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"}
Nov 03 01:30:10.495036 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.494845    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" event={"ID":"27f34cf8-614c-4856-8711-5f980c6a6ebe","Type":"ContainerStarted","Data":"110c56016716a8074bd49c241cb094558842b86f31aedbe21e73f7fe904067bd"}
Nov 03 01:30:10.522634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.522574    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podStartSLOduration=53.522558784 podStartE2EDuration="53.522558784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:10.517121279 +0000 UTC m=+2955.730864996" watchObservedRunningTime="2026-11-03 01:30:10.522558784 +0000 UTC m=+2955.736302446"
Nov 03 01:30:10.712666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.712632    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:10.714374 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.714349    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:10.714479 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:10.714399    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:11.508325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:11.505494    4531 generic.go:356] "Generic (PLEG): container finished" podID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerID="153f11294e8bfbd5afea776f4c13ce8c9835d136416387803f61596d25e9ff77" exitCode=0
Nov 03 01:30:11.508325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:11.506513    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" event={"ID":"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba","Type":"ContainerDied","Data":"153f11294e8bfbd5afea776f4c13ce8c9835d136416387803f61596d25e9ff77"}
Nov 03 01:30:11.713084 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:11.713042    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:11.713274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:11.713106    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:12.122020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.121990    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 01:30:12.196287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.193140    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:30:12.196287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.194361    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy"
Nov 03 01:30:12.196287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.194380    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy"
Nov 03 01:30:12.196287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.194548    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" containerName="istio-proxy"
Nov 03 01:30:12.198501 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.198479    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.201993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.201974    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b4971a-serving-cert\""
Nov 03 01:30:12.203022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.202998    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:30:12.217881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.217854    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218074    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218524    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218656 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218595    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:12.218656 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218636    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218818 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218686    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218818 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218718    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.218818 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.219047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218892    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.219047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218897    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:12.219047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.218988    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\") pod \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\" (UID: \"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba\") "
Nov 03 01:30:12.219941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.219773    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.219941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.219791    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:12.219941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.219801    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.220234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.220031    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:12.220234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.220214    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:12.222771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.222749    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "kube-api-access-vvf9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:12.223999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.223977    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:12.224410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.224384    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:12.224504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.224467    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy" pod "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" (UID: "7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:12.320866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.320818    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.320866 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.320869    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whv94\" (UniqueName: \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.321127 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.320973    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.321350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321321    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.321460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321447    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321463    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvf9h\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-kube-api-access-vvf9h\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321474    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321485    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321500    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321538    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.321668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.321554    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:12.423210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.423103    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whv94\" (UniqueName: \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.423381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.423233    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.423462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.423414    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.423524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.423475    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.424291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.424264    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.425876 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.425852    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.426316 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.425864    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.461182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.458527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whv94\" (UniqueName: \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\") pod \"maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.523350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.519234    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:12.620104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.620076    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"
Nov 03 01:30:12.620280 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.620098    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv" event={"ID":"7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba","Type":"ContainerDied","Data":"da0bcbc5b14d2fec1d30b4933b7caa1c8bf2c4f27c1fca1e3d5554055e9ecd62"}
Nov 03 01:30:12.620280 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.620225    4531 scope.go:122] "RemoveContainer" containerID="153f11294e8bfbd5afea776f4c13ce8c9835d136416387803f61596d25e9ff77"
Nov 03 01:30:12.693044 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.692985    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 01:30:12.701467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.700118    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-r2wbv"]
Nov 03 01:30:12.712191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.712135    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:12.712346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.712228    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:12.906481 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.906448    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba" path="/var/lib/kubelet/pods/7baf8d5c-f79f-4de8-b8a6-2e40e096c5ba/volumes"
Nov 03 01:30:12.967822 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:12.967789    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a53354e_580b_4411_98d1_46e59aa3ab56.slice/crio-51900d5d6315e96ea8d4e1b207e0d28cf7f7d1521653df22da12cf70b8f5d650 WatchSource:0}: Error finding container 51900d5d6315e96ea8d4e1b207e0d28cf7f7d1521653df22da12cf70b8f5d650: Status 404 returned error can't find the container with id 51900d5d6315e96ea8d4e1b207e0d28cf7f7d1521653df22da12cf70b8f5d650
Nov 03 01:30:12.969497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.969475    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:30:12.972017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:12.971984    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:30:13.666187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:13.658823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" event={"ID":"3a53354e-580b-4411-98d1-46e59aa3ab56","Type":"ContainerStarted","Data":"51900d5d6315e96ea8d4e1b207e0d28cf7f7d1521653df22da12cf70b8f5d650"}
Nov 03 01:30:13.712564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:13.712516    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:13.712728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:13.712588    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:14.671229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:14.671189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" event={"ID":"3a53354e-580b-4411-98d1-46e59aa3ab56","Type":"ContainerStarted","Data":"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"}
Nov 03 01:30:14.671670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:14.671282    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:14.693303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:14.693226    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" podStartSLOduration=30.076060927 podStartE2EDuration="30.693207501s" totalImagesPullingTime="617.146574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:14.688541021 +0000 UTC m=+2959.902284699" watchObservedRunningTime="2026-11-03 01:30:14.693207501 +0000 UTC m=+2959.906951175"
Nov 03 01:30:14.712775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:14.712741    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:14.712940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:14.712794    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:15.089933 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:15.089892    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 01:30:15.090190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:15.090129    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy" containerID="cri-o://4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac" gracePeriod=30
Nov 03 01:30:15.714183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:15.712307    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:15.714183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:15.712368    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:16.713062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:16.713026    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:16.713375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:16.713090    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:16.907691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:16.907660    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:16.908106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:16.907974    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:17.232833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.232793    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:17.240521 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:17.240470    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb807a829_b4a3_4df2_ac05_9bf0601b4e11.slice/crio-c4644606814e6b0fc5077ec4f6c9d4b94f9df025ee25428d2559bb169b5f0f99 WatchSource:0}: Error finding container c4644606814e6b0fc5077ec4f6c9d4b94f9df025ee25428d2559bb169b5f0f99: Status 404 returned error can't find the container with id c4644606814e6b0fc5077ec4f6c9d4b94f9df025ee25428d2559bb169b5f0f99
Nov 03 01:30:17.713121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.713086    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:30:17.713347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.713142    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:30:17.735607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.735570    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerStarted","Data":"c4644606814e6b0fc5077ec4f6c9d4b94f9df025ee25428d2559bb169b5f0f99"}
Nov 03 01:30:17.917025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.910417    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:17.917025 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:17.910908    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:18.209363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.209332    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:18.211318 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:18.211287    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1063ff_cc0e_43fd_ace6_348f52a17941.slice/crio-5136b9ccd41bbdc869dea142a214ac1bf522f8e45eca40ce68d4e11c8476eb8f WatchSource:0}: Error finding container 5136b9ccd41bbdc869dea142a214ac1bf522f8e45eca40ce68d4e11c8476eb8f: Status 404 returned error can't find the container with id 5136b9ccd41bbdc869dea142a214ac1bf522f8e45eca40ce68d4e11c8476eb8f
Nov 03 01:30:18.747224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.747120    4531 generic.go:356] "Generic (PLEG): container finished" podID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerID="68bd79cb40745cf22c45d5041e40f4ff30bf5cb420b33a8fabf69f59260f69ea" exitCode=60
Nov 03 01:30:18.747224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.747201    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerDied","Data":"68bd79cb40745cf22c45d5041e40f4ff30bf5cb420b33a8fabf69f59260f69ea"}
Nov 03 01:30:18.747713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.747687    4531 scope.go:122] "RemoveContainer" containerID="68bd79cb40745cf22c45d5041e40f4ff30bf5cb420b33a8fabf69f59260f69ea"
Nov 03 01:30:18.749139 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.749109    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerStarted","Data":"c7ce093ba1bdda968b56a8e974231a5954c5c36a77057a494e0441173fdbd186"}
Nov 03 01:30:18.749249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.749211    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerStarted","Data":"5136b9ccd41bbdc869dea142a214ac1bf522f8e45eca40ce68d4e11c8476eb8f"}
Nov 03 01:30:18.790494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.790430    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podStartSLOduration=18.790413057 podStartE2EDuration="18.790413057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:18.783782185 +0000 UTC m=+2963.997525857" watchObservedRunningTime="2026-11-03 01:30:18.790413057 +0000 UTC m=+2964.004156730"
Nov 03 01:30:18.896409 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.896361    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:18.897037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:18.897005    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:19.111458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.111406    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:19.112608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.112579    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:19.113815 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.113794    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:30:19.222706 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:19.222557    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64550299_3851_43d1_b2ac_4a2ddb450ac7.slice/crio-443a67a94a79fc71f1e04dc5e96d547946a8b4855b60c9f028f6598189b2bab1 WatchSource:0}: Error finding container 443a67a94a79fc71f1e04dc5e96d547946a8b4855b60c9f028f6598189b2bab1: Status 404 returned error can't find the container with id 443a67a94a79fc71f1e04dc5e96d547946a8b4855b60c9f028f6598189b2bab1
Nov 03 01:30:19.222897 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.222873    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:19.758334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.758298    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerStarted","Data":"9173bc6018bbeef27e242e7fa0a8fd85248cbb611a78a342d602fb5b45869d8f"}
Nov 03 01:30:19.758493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.758363    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerStarted","Data":"443a67a94a79fc71f1e04dc5e96d547946a8b4855b60c9f028f6598189b2bab1"}
Nov 03 01:30:19.760611 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.760585    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerStarted","Data":"a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"}
Nov 03 01:30:19.762513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.762488    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerID="c7ce093ba1bdda968b56a8e974231a5954c5c36a77057a494e0441173fdbd186" exitCode=60
Nov 03 01:30:19.762623 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.762576    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerDied","Data":"c7ce093ba1bdda968b56a8e974231a5954c5c36a77057a494e0441173fdbd186"}
Nov 03 01:30:19.762876 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.762858    4531 scope.go:122] "RemoveContainer" containerID="c7ce093ba1bdda968b56a8e974231a5954c5c36a77057a494e0441173fdbd186"
Nov 03 01:30:19.777803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.777734    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podStartSLOduration=19.777721472 podStartE2EDuration="19.777721472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:19.771916482 +0000 UTC m=+2964.985660155" watchObservedRunningTime="2026-11-03 01:30:19.777721472 +0000 UTC m=+2964.991465143"
Nov 03 01:30:19.811718 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.807857    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podStartSLOduration=19.807840024 podStartE2EDuration="19.807840024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:19.806991677 +0000 UTC m=+2965.020735350" watchObservedRunningTime="2026-11-03 01:30:19.807840024 +0000 UTC m=+2965.021583698"
Nov 03 01:30:19.895837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.895803    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:19.896365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:19.896346    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:20.284531 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:20.284495    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f53f02_45ad_44bd_bf1e_f519f5332df2.slice/crio-ccc8bc3267800093f71d00aa5b830c84ed0da94eda6b817f99ea3c936da6b352 WatchSource:0}: Error finding container ccc8bc3267800093f71d00aa5b830c84ed0da94eda6b817f99ea3c936da6b352: Status 404 returned error can't find the container with id ccc8bc3267800093f71d00aa5b830c84ed0da94eda6b817f99ea3c936da6b352
Nov 03 01:30:20.286474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.286451    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:20.388474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.388441    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 01:30:20.411252 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411208    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411299    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411535 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411411    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411554    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411645    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411690    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411718    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.411822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.411791    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\") pod \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\" (UID: \"ad49df2b-f905-40c4-aa39-cf966d5dbc25\") "
Nov 03 01:30:20.412757 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.412729    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:20.412878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.412857    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:20.413068 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.413049    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:20.413303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.413284    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:20.413598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.413388    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:20.415692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.415657    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "kube-api-access-m4s2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:20.416206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.416128    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:20.416362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.416221    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:20.417287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.417261    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo" pod "ad49df2b-f905-40c4-aa39-cf966d5dbc25" (UID: "ad49df2b-f905-40c4-aa39-cf966d5dbc25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:20.425966 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.425837    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:20.428964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.427042    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy"
Nov 03 01:30:20.428964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.427061    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy"
Nov 03 01:30:20.428964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.427209    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerName="istio-proxy"
Nov 03 01:30:20.441365 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.441340    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:20.442956 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.442928    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.450216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.450195    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.454941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.453800    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:20.486877 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.486147    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.493837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.493809    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:20.505930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.505899    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.514040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514012    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj896\" (UniqueName: \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.514306 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514129    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmsh6\" (UniqueName: \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.514306 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514238    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vksd9\" (UniqueName: \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.514438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514311    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.514438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514336    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.514559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514444    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.514639 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514621    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514646    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4s2l\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-kube-api-access-m4s2l\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514662    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514675    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514688    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514701    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514713    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514724    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.514946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.514736    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad49df2b-f905-40c4-aa39-cf966d5dbc25-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:20.552301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.552221    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:20.566102 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.566077    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.572717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.571473    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:20.587073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.587041    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:20.592063 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.588434    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.614617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.614582    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:20.614777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.614726    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.616110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vksd9\" (UniqueName: \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.616259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.616259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616223    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.616259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616251    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5qv\" (UniqueName: \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.616444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616372    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.616444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjl8k\" (UniqueName: \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.616554 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616461    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.616610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616559    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wfb\" (UniqueName: \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.616665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj896\" (UniqueName: \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.617024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616713    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmsh6\" (UniqueName: \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.617024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616747    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.617024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.616797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.620792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.620366    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.620792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.620639    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.621113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.621000    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.629834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.629816    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:20.630197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.630069    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.641277 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.641254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj896\" (UniqueName: \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\") pod \"maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.642622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.642035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmsh6\" (UniqueName: \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\") pod \"maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.642758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.642567    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vksd9\" (UniqueName: \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\") pod \"maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.646886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.646859    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:20.648474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.647066    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.662854 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.662834    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.662974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.662960    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:20.663043 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.663003    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:20.663043 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.663026    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:20.663148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.663054    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:20.695328 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.694648    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:20.702761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.698144    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:20.709594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.707554    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:20.713311 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.713268    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:20.715762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.715742    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:30:20.718006 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.717978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjl8k\" (UniqueName: \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.718105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.718105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718070    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k55h\" (UniqueName: \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.718230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718129    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqnc\" (UniqueName: \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.718230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718211    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wfb\" (UniqueName: \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.718349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718302    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7v4v\" (UniqueName: \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.718401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718351    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.718401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718388    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.718498 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718424    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.718549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718511    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wq4\" (UniqueName: \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.718603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.718603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718594    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5qv\" (UniqueName: \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.718708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.718708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.718655    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.719655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.719628    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.720174 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.720143    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:20.720460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.720438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.720939 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.720918    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.728649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.728269    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wfb\" (UniqueName: \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.728938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.728892    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5qv\" (UniqueName: \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\") pod \"maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.729059 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.729038    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjl8k\" (UniqueName: \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.732304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.732284    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:30:20.732802 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.732484    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy" containerID="cri-o://fb1d79abe62b243d142f19787536bda2f0057b09ff3ec6e4540f2518b2e07071" gracePeriod=30
Nov 03 01:30:20.749179 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.744931    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:20.785593 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.785555    4531 generic.go:356] "Generic (PLEG): container finished" podID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerID="9173bc6018bbeef27e242e7fa0a8fd85248cbb611a78a342d602fb5b45869d8f" exitCode=60
Nov 03 01:30:20.785782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.785694    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerDied","Data":"9173bc6018bbeef27e242e7fa0a8fd85248cbb611a78a342d602fb5b45869d8f"}
Nov 03 01:30:20.786312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.786286    4531 scope.go:122] "RemoveContainer" containerID="9173bc6018bbeef27e242e7fa0a8fd85248cbb611a78a342d602fb5b45869d8f"
Nov 03 01:30:20.788758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.788728    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerStarted","Data":"ccc8bc3267800093f71d00aa5b830c84ed0da94eda6b817f99ea3c936da6b352"}
Nov 03 01:30:20.791402 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.791374    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" containerID="4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac" exitCode=0
Nov 03 01:30:20.791494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.791482    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" event={"ID":"ad49df2b-f905-40c4-aa39-cf966d5dbc25","Type":"ContainerDied","Data":"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"}
Nov 03 01:30:20.791566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.791508    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"
Nov 03 01:30:20.791566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.791524    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5" event={"ID":"ad49df2b-f905-40c4-aa39-cf966d5dbc25","Type":"ContainerDied","Data":"8f90f345bfe89ed887a6b7e100d028edde17439d27596a355b0109cd3b561b47"}
Nov 03 01:30:20.791566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.791556    4531 scope.go:122] "RemoveContainer" containerID="4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"
Nov 03 01:30:20.793802 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.793619    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:20.799670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.798579    4531 generic.go:356] "Generic (PLEG): container finished" podID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerID="a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264" exitCode=60
Nov 03 01:30:20.799670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.798654    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerDied","Data":"a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"}
Nov 03 01:30:20.799670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.799109    4531 scope.go:122] "RemoveContainer" containerID="a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"
Nov 03 01:30:20.799670 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:20.799449    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11"
Nov 03 01:30:20.809062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.809033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerStarted","Data":"cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"}
Nov 03 01:30:20.812327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.812299    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:20.821042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.820969    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k55h\" (UniqueName: \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.821183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821060    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqnc\" (UniqueName: \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7v4v\" (UniqueName: \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821325    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821460    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wq4\" (UniqueName: \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.821911 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.821614    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.822299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.822195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.823417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.822825    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.823417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.823372    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.823417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.823404    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.826706 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.826624    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:20.846803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.846771    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k55h\" (UniqueName: \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\") pod \"maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:20.848081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.847467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqnc\" (UniqueName: \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:20.848399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.848114    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7v4v\" (UniqueName: \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:20.848791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.848766    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wq4\" (UniqueName: \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:20.888714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.888687    4531 scope.go:122] "RemoveContainer" containerID="4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"
Nov 03 01:30:20.889282 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:20.889140    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac\": container with ID starting with 4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac not found: ID does not exist" containerID="4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"
Nov 03 01:30:20.889809 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.889446    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac"} err="failed to get container status \"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac\": rpc error: code = NotFound desc = could not find container \"4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac\": container with ID starting with 4f4c5e478e0563faf7f4b242d7c5195e8ea1dbd933f8a31ec8bbd5f4a16510ac not found: ID does not exist"
Nov 03 01:30:20.889809 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.889481    4531 scope.go:122] "RemoveContainer" containerID="68bd79cb40745cf22c45d5041e40f4ff30bf5cb420b33a8fabf69f59260f69ea"
Nov 03 01:30:20.890965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.890952    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 01:30:20.894923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.894901    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-rk5k5"]
Nov 03 01:30:20.897923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.897902    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:20.910295 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.909833    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad49df2b-f905-40c4-aa39-cf966d5dbc25" path="/var/lib/kubelet/pods/ad49df2b-f905-40c4-aa39-cf966d5dbc25/volumes"
Nov 03 01:30:20.931210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.930874    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:20.937121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.934773    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:20.974571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:20.974534    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:21.008853 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.008618    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:21.022520 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.022320    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:21.047033 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.046947    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:21.273073 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.273020    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:21.429286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.428649    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:21.436399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.436370    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:21.512210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.512175    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:21.516940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.516914    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:21.534596 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:21.534550    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b22bb9_80bf_460b_836e_2e8af3b30d83.slice/crio-824fb6cce7b7b27406eb311bcc09feca7676bd9cc5d9f654bf6a5184f0643311 WatchSource:0}: Error finding container 824fb6cce7b7b27406eb311bcc09feca7676bd9cc5d9f654bf6a5184f0643311: Status 404 returned error can't find the container with id 824fb6cce7b7b27406eb311bcc09feca7676bd9cc5d9f654bf6a5184f0643311
Nov 03 01:30:21.699423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.699392    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:21.699838 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:21.699812    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdc6cdc_b2f6_4e3e_b02e_a985ce357fa8.slice/crio-c4f8580216e5cded786146683eb5aa6004f9e9a686c066be14ba96fa9eabc046 WatchSource:0}: Error finding container c4f8580216e5cded786146683eb5aa6004f9e9a686c066be14ba96fa9eabc046: Status 404 returned error can't find the container with id c4f8580216e5cded786146683eb5aa6004f9e9a686c066be14ba96fa9eabc046
Nov 03 01:30:21.763236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.763197    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:21.782493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.782430    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:21.787803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.787779    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:21.811702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.811674    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:21.817055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.817026    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerStarted","Data":"824fb6cce7b7b27406eb311bcc09feca7676bd9cc5d9f654bf6a5184f0643311"}
Nov 03 01:30:21.818488 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.818460    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerStarted","Data":"7dfc2cdba31369219af4345c2e94ec45d258adea4b208bc07879fe025af98358"}
Nov 03 01:30:21.819740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.819719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerStarted","Data":"d86dd242cb98c2ca84cc11bab8341eaf6772fbb8ce3b69eb0672479cc9bfff0d"}
Nov 03 01:30:21.821010 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.820986    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerStarted","Data":"a0ded55a9af8093bb1086b3f6b84ab00c73c573c47a1c327739989b9db7f3b0f"}
Nov 03 01:30:21.822961 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.822935    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerStarted","Data":"d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"}
Nov 03 01:30:21.826564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.826546    4531 scope.go:122] "RemoveContainer" containerID="a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"
Nov 03 01:30:21.827386 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:21.827360    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11"
Nov 03 01:30:21.831760 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.828921    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerID="cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902" exitCode=60
Nov 03 01:30:21.831760 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.828994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerDied","Data":"cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"}
Nov 03 01:30:21.831760 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.829036    4531 scope.go:122] "RemoveContainer" containerID="c7ce093ba1bdda968b56a8e974231a5954c5c36a77057a494e0441173fdbd186"
Nov 03 01:30:21.831760 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.829527    4531 scope.go:122] "RemoveContainer" containerID="cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"
Nov 03 01:30:21.831760 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:21.829935    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941"
Nov 03 01:30:21.832392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.832370    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerStarted","Data":"c4f8580216e5cded786146683eb5aa6004f9e9a686c066be14ba96fa9eabc046"}
Nov 03 01:30:21.835613 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.835347    4531 generic.go:356] "Generic (PLEG): container finished" podID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerID="0e1bbed321099739259b62f6c7d3afd158b03425e869f5c1c981381b88df5f0e" exitCode=60
Nov 03 01:30:21.835717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.835681    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerDied","Data":"0e1bbed321099739259b62f6c7d3afd158b03425e869f5c1c981381b88df5f0e"}
Nov 03 01:30:21.836062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.836044    4531 scope.go:122] "RemoveContainer" containerID="0e1bbed321099739259b62f6c7d3afd158b03425e869f5c1c981381b88df5f0e"
Nov 03 01:30:21.837575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:21.837549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerStarted","Data":"129b6dca9572eec9e80c917419a3a5016fb20be328a99f2327fcd344e65e282f"}
Nov 03 01:30:21.937815 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:21.937780    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b335e2_51de_4e95_85f2_75b256c50838.slice/crio-9918c469526c98f36e7fa825bbf78e4be1957fad09f051b99e0d0f43036d84a5 WatchSource:0}: Error finding container 9918c469526c98f36e7fa825bbf78e4be1957fad09f051b99e0d0f43036d84a5: Status 404 returned error can't find the container with id 9918c469526c98f36e7fa825bbf78e4be1957fad09f051b99e0d0f43036d84a5
Nov 03 01:30:21.940986 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:21.940958    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934cfef1_1346_41b4_b41c_e0ec6b27aaee.slice/crio-f429e3d2f654950d4b3fe201558eed1d0ab82a03a045596fb35838bf31cabcb9 WatchSource:0}: Error finding container f429e3d2f654950d4b3fe201558eed1d0ab82a03a045596fb35838bf31cabcb9: Status 404 returned error can't find the container with id f429e3d2f654950d4b3fe201558eed1d0ab82a03a045596fb35838bf31cabcb9
Nov 03 01:30:22.099184 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:22.082280    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408dcbcb_fb13_4ccf_a936_8d486fcc11f5.slice/crio-71e9909036b97851aedfb4b6f6a1961361df2b351e6b8034ece8ab067255d7c5 WatchSource:0}: Error finding container 71e9909036b97851aedfb4b6f6a1961361df2b351e6b8034ece8ab067255d7c5: Status 404 returned error can't find the container with id 71e9909036b97851aedfb4b6f6a1961361df2b351e6b8034ece8ab067255d7c5
Nov 03 01:30:22.853766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.853704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerStarted","Data":"9918c469526c98f36e7fa825bbf78e4be1957fad09f051b99e0d0f43036d84a5"}
Nov 03 01:30:22.855654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.855624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerStarted","Data":"e0a4f43ce38c2b319070b8b584808eb64d401db58035b1e4a56acb9664b90e81"}
Nov 03 01:30:22.859259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.858130    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerStarted","Data":"6a44ff64ed3d70d9bee688256cbc96493e9f04e0cb1dcfbb3d860b816a2f3cc9"}
Nov 03 01:30:22.860670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.860646    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerStarted","Data":"fff5893a293137d3ba2f2ab6c14f5062371c9a886d4952abd3beb853e3e6ba4e"}
Nov 03 01:30:22.863144 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.863117    4531 generic.go:356] "Generic (PLEG): container finished" podID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerID="d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e" exitCode=60
Nov 03 01:30:22.863308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.863271    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerDied","Data":"d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"}
Nov 03 01:30:22.863371 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.863315    4531 scope.go:122] "RemoveContainer" containerID="9173bc6018bbeef27e242e7fa0a8fd85248cbb611a78a342d602fb5b45869d8f"
Nov 03 01:30:22.863474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.863455    4531 scope.go:122] "RemoveContainer" containerID="d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"
Nov 03 01:30:22.863834 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:22.863810    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7"
Nov 03 01:30:22.865550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.865495    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerStarted","Data":"71e9909036b97851aedfb4b6f6a1961361df2b351e6b8034ece8ab067255d7c5"}
Nov 03 01:30:22.867758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.867737    4531 scope.go:122] "RemoveContainer" containerID="cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"
Nov 03 01:30:22.868049 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:22.868027    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941"
Nov 03 01:30:22.870427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.870402    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerStarted","Data":"c50035bcb40d128e0a53bc243be9c7a2a3caa137109fe99133625f8e114153da"}
Nov 03 01:30:22.874590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.873295    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerStarted","Data":"1f29fdb2c0ec437c519d448b2e4b447b4a865284f588445269e004bd798b7a40"}
Nov 03 01:30:22.875542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.875495    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" podStartSLOduration=22.875482039 podStartE2EDuration="22.875482039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:22.875294861 +0000 UTC m=+2968.089038534" watchObservedRunningTime="2026-11-03 01:30:22.875482039 +0000 UTC m=+2968.089225713"
Nov 03 01:30:22.877096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.876619    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerStarted","Data":"c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"}
Nov 03 01:30:22.878589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.878550    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerStarted","Data":"8a34d95a69568a8da93ba6aabb46bd4535e875f01ba0c2ff51029df6f855d330"}
Nov 03 01:30:22.880712 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.880692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerStarted","Data":"f429e3d2f654950d4b3fe201558eed1d0ab82a03a045596fb35838bf31cabcb9"}
Nov 03 01:30:22.882696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.882675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerStarted","Data":"d1b0b7ec0cc860e04df2cfdff6d5a81795ef1a9d7c8ea2632403de4feed1dabb"}
Nov 03 01:30:22.899269 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.899230    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" podStartSLOduration=22.899216553 podStartE2EDuration="22.899216553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:22.894603855 +0000 UTC m=+2968.108347540" watchObservedRunningTime="2026-11-03 01:30:22.899216553 +0000 UTC m=+2968.112960224"
Nov 03 01:30:22.997598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:22.995033    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" podStartSLOduration=22.995013283 podStartE2EDuration="22.995013283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:22.923750261 +0000 UTC m=+2968.137493933" watchObservedRunningTime="2026-11-03 01:30:22.995013283 +0000 UTC m=+2968.208756957"
Nov 03 01:30:23.034407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.033412    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" podStartSLOduration=23.033396473 podStartE2EDuration="23.033396473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.01454376 +0000 UTC m=+2968.228287434" watchObservedRunningTime="2026-11-03 01:30:23.033396473 +0000 UTC m=+2968.247140147"
Nov 03 01:30:23.036963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.036602    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" podStartSLOduration=23.036590137 podStartE2EDuration="23.036590137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:22.992669839 +0000 UTC m=+2968.206413507" watchObservedRunningTime="2026-11-03 01:30:23.036590137 +0000 UTC m=+2968.250333810"
Nov 03 01:30:23.066634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.066585    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" podStartSLOduration=23.066570193 podStartE2EDuration="23.066570193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.05269 +0000 UTC m=+2968.266433673" watchObservedRunningTime="2026-11-03 01:30:23.066570193 +0000 UTC m=+2968.280313866"
Nov 03 01:30:23.088722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.088662    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" podStartSLOduration=23.088644273 podStartE2EDuration="23.088644273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.084412857 +0000 UTC m=+2968.298156530" watchObservedRunningTime="2026-11-03 01:30:23.088644273 +0000 UTC m=+2968.302387947"
Nov 03 01:30:23.105696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.105648    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podStartSLOduration=23.10560616 podStartE2EDuration="23.10560616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.100710015 +0000 UTC m=+2968.314453691" watchObservedRunningTime="2026-11-03 01:30:23.10560616 +0000 UTC m=+2968.319349874"
Nov 03 01:30:23.121174 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.121124    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" podStartSLOduration=23.121110855 podStartE2EDuration="23.121110855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:23.117948624 +0000 UTC m=+2968.331692297" watchObservedRunningTime="2026-11-03 01:30:23.121110855 +0000 UTC m=+2968.334854528"
Nov 03 01:30:23.896044 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.895945    4531 generic.go:356] "Generic (PLEG): container finished" podID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerID="6a44ff64ed3d70d9bee688256cbc96493e9f04e0cb1dcfbb3d860b816a2f3cc9" exitCode=60
Nov 03 01:30:23.896510 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.896029    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerDied","Data":"6a44ff64ed3d70d9bee688256cbc96493e9f04e0cb1dcfbb3d860b816a2f3cc9"}
Nov 03 01:30:23.896510 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.896439    4531 scope.go:122] "RemoveContainer" containerID="6a44ff64ed3d70d9bee688256cbc96493e9f04e0cb1dcfbb3d860b816a2f3cc9"
Nov 03 01:30:23.898156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.898130    4531 generic.go:356] "Generic (PLEG): container finished" podID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerID="fff5893a293137d3ba2f2ab6c14f5062371c9a886d4952abd3beb853e3e6ba4e" exitCode=60
Nov 03 01:30:23.898292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.898269    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerDied","Data":"fff5893a293137d3ba2f2ab6c14f5062371c9a886d4952abd3beb853e3e6ba4e"}
Nov 03 01:30:23.898504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.898488    4531 scope.go:122] "RemoveContainer" containerID="fff5893a293137d3ba2f2ab6c14f5062371c9a886d4952abd3beb853e3e6ba4e"
Nov 03 01:30:23.900875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.900777    4531 scope.go:122] "RemoveContainer" containerID="d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"
Nov 03 01:30:23.901055 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:23.901029    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7"
Nov 03 01:30:23.903114 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.903092    4531 generic.go:356] "Generic (PLEG): container finished" podID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerID="f1691ca5ecee440edc7ba303be07a42162028f3e934b0ab78a4194c4c863ca5f" exitCode=60
Nov 03 01:30:23.903252 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.903227    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerDied","Data":"f1691ca5ecee440edc7ba303be07a42162028f3e934b0ab78a4194c4c863ca5f"}
Nov 03 01:30:23.903389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.903370    4531 scope.go:122] "RemoveContainer" containerID="f1691ca5ecee440edc7ba303be07a42162028f3e934b0ab78a4194c4c863ca5f"
Nov 03 01:30:23.905080 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.905061    4531 generic.go:356] "Generic (PLEG): container finished" podID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerID="c50035bcb40d128e0a53bc243be9c7a2a3caa137109fe99133625f8e114153da" exitCode=60
Nov 03 01:30:23.905188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.905146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerDied","Data":"c50035bcb40d128e0a53bc243be9c7a2a3caa137109fe99133625f8e114153da"}
Nov 03 01:30:23.905334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.905320    4531 scope.go:122] "RemoveContainer" containerID="c50035bcb40d128e0a53bc243be9c7a2a3caa137109fe99133625f8e114153da"
Nov 03 01:30:23.907062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.907040    4531 generic.go:356] "Generic (PLEG): container finished" podID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerID="a01faec10b0623ba548be213824919718a75e71e26aafef1a17d4db6ae4ef747" exitCode=60
Nov 03 01:30:23.907133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.907109    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerDied","Data":"a01faec10b0623ba548be213824919718a75e71e26aafef1a17d4db6ae4ef747"}
Nov 03 01:30:23.907322 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.907308    4531 scope.go:122] "RemoveContainer" containerID="a01faec10b0623ba548be213824919718a75e71e26aafef1a17d4db6ae4ef747"
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.909233    4531 generic.go:356] "Generic (PLEG): container finished" podID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerID="c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948" exitCode=60
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.909530    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerDied","Data":"c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"}
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.909572    4531 scope.go:122] "RemoveContainer" containerID="0e1bbed321099739259b62f6c7d3afd158b03425e869f5c1c981381b88df5f0e"
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.909894    4531 scope.go:122] "RemoveContainer" containerID="c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:23.910170    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2"
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.911420    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerID="8a34d95a69568a8da93ba6aabb46bd4535e875f01ba0c2ff51029df6f855d330" exitCode=60
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.911505    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerDied","Data":"8a34d95a69568a8da93ba6aabb46bd4535e875f01ba0c2ff51029df6f855d330"}
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.911659    4531 scope.go:122] "RemoveContainer" containerID="8a34d95a69568a8da93ba6aabb46bd4535e875f01ba0c2ff51029df6f855d330"
Nov 03 01:30:23.913513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.913492    4531 generic.go:356] "Generic (PLEG): container finished" podID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerID="72d90daf3b1f126dab2d8fc6e6992c1fb3591ffe6864d9b1c9df96f85d6f6303" exitCode=60
Nov 03 01:30:23.914368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.913697    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerDied","Data":"72d90daf3b1f126dab2d8fc6e6992c1fb3591ffe6864d9b1c9df96f85d6f6303"}
Nov 03 01:30:23.914368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.914143    4531 scope.go:122] "RemoveContainer" containerID="72d90daf3b1f126dab2d8fc6e6992c1fb3591ffe6864d9b1c9df96f85d6f6303"
Nov 03 01:30:23.915976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.915955    4531 generic.go:356] "Generic (PLEG): container finished" podID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerID="d1b0b7ec0cc860e04df2cfdff6d5a81795ef1a9d7c8ea2632403de4feed1dabb" exitCode=60
Nov 03 01:30:23.916069 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.916057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerDied","Data":"d1b0b7ec0cc860e04df2cfdff6d5a81795ef1a9d7c8ea2632403de4feed1dabb"}
Nov 03 01:30:23.916453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.916434    4531 scope.go:122] "RemoveContainer" containerID="d1b0b7ec0cc860e04df2cfdff6d5a81795ef1a9d7c8ea2632403de4feed1dabb"
Nov 03 01:30:23.918272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.918251    4531 generic.go:356] "Generic (PLEG): container finished" podID="26b335e2-51de-4e95-85f2-75b256c50838" containerID="92ea76b4159ba0388942ef4391beb526186e554b94b8cc1f3620746980ac244c" exitCode=60
Nov 03 01:30:23.918464 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.918308    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerDied","Data":"92ea76b4159ba0388942ef4391beb526186e554b94b8cc1f3620746980ac244c"}
Nov 03 01:30:23.918568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.918537    4531 scope.go:122] "RemoveContainer" containerID="92ea76b4159ba0388942ef4391beb526186e554b94b8cc1f3620746980ac244c"
Nov 03 01:30:23.920526 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.920501    4531 generic.go:356] "Generic (PLEG): container finished" podID="04b00673-3454-4536-bc2b-571820696dbc" containerID="e0a4f43ce38c2b319070b8b584808eb64d401db58035b1e4a56acb9664b90e81" exitCode=60
Nov 03 01:30:23.920611 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.920547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerDied","Data":"e0a4f43ce38c2b319070b8b584808eb64d401db58035b1e4a56acb9664b90e81"}
Nov 03 01:30:23.920832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:23.920811    4531 scope.go:122] "RemoveContainer" containerID="e0a4f43ce38c2b319070b8b584808eb64d401db58035b1e4a56acb9664b90e81"
Nov 03 01:30:24.938800 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.938767    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerStarted","Data":"4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"}
Nov 03 01:30:24.941256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.941221    4531 scope.go:122] "RemoveContainer" containerID="c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"
Nov 03 01:30:24.941670 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:24.941639    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2"
Nov 03 01:30:24.943208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.943135    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerStarted","Data":"f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"}
Nov 03 01:30:24.945425 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.945385    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerStarted","Data":"b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"}
Nov 03 01:30:24.947509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.947481    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerStarted","Data":"ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"}
Nov 03 01:30:24.967083 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.967045    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerStarted","Data":"0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"}
Nov 03 01:30:24.970667 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.970636    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerStarted","Data":"aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"}
Nov 03 01:30:24.975852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.975820    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerStarted","Data":"5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"}
Nov 03 01:30:24.978566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.978540    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerStarted","Data":"63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"}
Nov 03 01:30:24.983069 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.983036    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerStarted","Data":"c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"}
Nov 03 01:30:24.994265 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:24.994106    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerStarted","Data":"24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"}
Nov 03 01:30:25.029135 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:25.029070    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" podStartSLOduration=25.029051383 podStartE2EDuration="25.029051383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:25.018337979 +0000 UTC m=+2970.232081652" watchObservedRunningTime="2026-11-03 01:30:25.029051383 +0000 UTC m=+2970.242795061"
Nov 03 01:30:25.078623 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:25.078561    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" podStartSLOduration=25.078544049 podStartE2EDuration="25.078544049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:25.073793093 +0000 UTC m=+2970.287536765" watchObservedRunningTime="2026-11-03 01:30:25.078544049 +0000 UTC m=+2970.292287722"
Nov 03 01:30:26.010940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.010837    4531 generic.go:356] "Generic (PLEG): container finished" podID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerID="c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437" exitCode=60
Nov 03 01:30:26.011435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.010925    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerDied","Data":"c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"}
Nov 03 01:30:26.011435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.011007    4531 scope.go:122] "RemoveContainer" containerID="f1691ca5ecee440edc7ba303be07a42162028f3e934b0ab78a4194c4c863ca5f"
Nov 03 01:30:26.011435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.011319    4531 scope.go:122] "RemoveContainer" containerID="c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"
Nov 03 01:30:26.011753 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.011732    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f_odh-ai-gateway-infra(408dcbcb-fb13-4ccf-a936-8d486fcc11f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5"
Nov 03 01:30:26.015862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.015723    4531 generic.go:356] "Generic (PLEG): container finished" podID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerID="24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0" exitCode=60
Nov 03 01:30:26.015862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.015778    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerDied","Data":"24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"}
Nov 03 01:30:26.016217 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.016185    4531 scope.go:122] "RemoveContainer" containerID="24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"
Nov 03 01:30:26.016645 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.016616    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v_odh-ai-gateway-infra(3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8"
Nov 03 01:30:26.019002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.018975    4531 generic.go:356] "Generic (PLEG): container finished" podID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerID="4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db" exitCode=60
Nov 03 01:30:26.019118 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.019081    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerDied","Data":"4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"}
Nov 03 01:30:26.020942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.020920    4531 scope.go:122] "RemoveContainer" containerID="4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"
Nov 03 01:30:26.021397 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.021372    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg_odh-ai-gateway-infra(b3f6310c-f0ea-44c2-8109-7ba8d241b3f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1"
Nov 03 01:30:26.024278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.023147    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerID="f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850" exitCode=60
Nov 03 01:30:26.024278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.023246    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerDied","Data":"f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"}
Nov 03 01:30:26.024278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.023601    4531 scope.go:122] "RemoveContainer" containerID="f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"
Nov 03 01:30:26.024278 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.023910    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv_odh-ai-gateway-infra(2d50733f-a18e-43c7-9035-f0c331fd5511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511"
Nov 03 01:30:26.028610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.028504    4531 generic.go:356] "Generic (PLEG): container finished" podID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerID="b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666" exitCode=60
Nov 03 01:30:26.028801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.028741    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerDied","Data":"b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"}
Nov 03 01:30:26.028903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.028880    4531 scope.go:122] "RemoveContainer" containerID="b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"
Nov 03 01:30:26.029442 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.029384    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj_odh-ai-gateway-infra(934cfef1-1346-41b4-b41c-e0ec6b27aaee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee"
Nov 03 01:30:26.034357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.034333    4531 generic.go:356] "Generic (PLEG): container finished" podID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerID="ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c" exitCode=60
Nov 03 01:30:26.034470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.034454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerDied","Data":"ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"}
Nov 03 01:30:26.034875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.034856    4531 scope.go:122] "RemoveContainer" containerID="ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"
Nov 03 01:30:26.036382 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.035397    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp_odh-ai-gateway-infra(12b22bb9-80bf-460b-836e-2e8af3b30d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83"
Nov 03 01:30:26.044149 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.044121    4531 generic.go:356] "Generic (PLEG): container finished" podID="26b335e2-51de-4e95-85f2-75b256c50838" containerID="0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d" exitCode=60
Nov 03 01:30:26.044279 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.044236    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerDied","Data":"0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"}
Nov 03 01:30:26.045290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.044674    4531 scope.go:122] "RemoveContainer" containerID="0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"
Nov 03 01:30:26.045290 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.045063    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm_odh-ai-gateway-infra(26b335e2-51de-4e95-85f2-75b256c50838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" podUID="26b335e2-51de-4e95-85f2-75b256c50838"
Nov 03 01:30:26.048661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.048632    4531 generic.go:356] "Generic (PLEG): container finished" podID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerID="fb1d79abe62b243d142f19787536bda2f0057b09ff3ec6e4540f2518b2e07071" exitCode=0
Nov 03 01:30:26.048754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.048717    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" event={"ID":"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf","Type":"ContainerDied","Data":"fb1d79abe62b243d142f19787536bda2f0057b09ff3ec6e4540f2518b2e07071"}
Nov 03 01:30:26.050759 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.050738    4531 generic.go:356] "Generic (PLEG): container finished" podID="04b00673-3454-4536-bc2b-571820696dbc" containerID="aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc" exitCode=60
Nov 03 01:30:26.050860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.050819    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerDied","Data":"aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"}
Nov 03 01:30:26.051217 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.051200    4531 scope.go:122] "RemoveContainer" containerID="aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"
Nov 03 01:30:26.051530 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.051507    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw_odh-ai-gateway-infra(04b00673-3454-4536-bc2b-571820696dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" podUID="04b00673-3454-4536-bc2b-571820696dbc"
Nov 03 01:30:26.055644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.054869    4531 generic.go:356] "Generic (PLEG): container finished" podID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerID="5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126" exitCode=60
Nov 03 01:30:26.055644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.054959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerDied","Data":"5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"}
Nov 03 01:30:26.056075 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.056052    4531 scope.go:122] "RemoveContainer" containerID="5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"
Nov 03 01:30:26.056483 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.056461    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8_odh-ai-gateway-infra(e4e12344-17f1-4dc7-bd43-176d0f99b2c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9"
Nov 03 01:30:26.058759 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.058736    4531 generic.go:356] "Generic (PLEG): container finished" podID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerID="63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098" exitCode=60
Nov 03 01:30:26.058867 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.058849    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerDied","Data":"63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"}
Nov 03 01:30:26.059129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.059111    4531 scope.go:122] "RemoveContainer" containerID="63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"
Nov 03 01:30:26.060347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.060324    4531 scope.go:122] "RemoveContainer" containerID="c50035bcb40d128e0a53bc243be9c7a2a3caa137109fe99133625f8e114153da"
Nov 03 01:30:26.060866 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:26.060813    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml_odh-ai-gateway-infra(3d4447f1-1a4d-4645-88f4-a424a6d94bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3"
Nov 03 01:30:26.070594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.070569    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:30:26.082020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.081996    4531 scope.go:122] "RemoveContainer" containerID="a01faec10b0623ba548be213824919718a75e71e26aafef1a17d4db6ae4ef747"
Nov 03 01:30:26.097221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.097197    4531 scope.go:122] "RemoveContainer" containerID="8a34d95a69568a8da93ba6aabb46bd4535e875f01ba0c2ff51029df6f855d330"
Nov 03 01:30:26.115505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.115482    4531 scope.go:122] "RemoveContainer" containerID="72d90daf3b1f126dab2d8fc6e6992c1fb3591ffe6864d9b1c9df96f85d6f6303"
Nov 03 01:30:26.132575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.132542    4531 scope.go:122] "RemoveContainer" containerID="d1b0b7ec0cc860e04df2cfdff6d5a81795ef1a9d7c8ea2632403de4feed1dabb"
Nov 03 01:30:26.152874 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.152799    4531 scope.go:122] "RemoveContainer" containerID="92ea76b4159ba0388942ef4391beb526186e554b94b8cc1f3620746980ac244c"
Nov 03 01:30:26.180398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.180374    4531 scope.go:122] "RemoveContainer" containerID="e0a4f43ce38c2b319070b8b584808eb64d401db58035b1e4a56acb9664b90e81"
Nov 03 01:30:26.190389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.188541    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:30:26.190389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.190033    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy"
Nov 03 01:30:26.190389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.190055    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy"
Nov 03 01:30:26.190389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.190369    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" containerName="istio-proxy"
Nov 03 01:30:26.192674 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.192648    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.192836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.192731    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.192836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.192792    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.194262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.193353    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:26.195094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.195061    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.196589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.196565    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:26.197062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.197040    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:26.197786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.197764    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.197884 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.197848    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.197943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.197925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.198012 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.197982    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.198306 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.198063    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.199064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.198798    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\") pod \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\" (UID: \"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf\") "
Nov 03 01:30:26.199212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.198414    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:26.199212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.198639    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:26.199810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.199788    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.199887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.199813    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.199887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.199829    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.199887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.199843    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.199887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.199857    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.203332 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.203312    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b4971a-dockercfg-lzk7n\""
Nov 03 01:30:26.203433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.203393    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b4971a\""
Nov 03 01:30:26.203433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.203427    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:30:26.213800 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.213730    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:26.222905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.222873    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "kube-api-access-s5wn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:26.223613 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.223579    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:26.224516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.224495    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token" pod "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" (UID: "bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:26.262040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.262016    4531 scope.go:122] "RemoveContainer" containerID="6a44ff64ed3d70d9bee688256cbc96493e9f04e0cb1dcfbb3d860b816a2f3cc9"
Nov 03 01:30:26.296967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.296939    4531 scope.go:122] "RemoveContainer" containerID="fff5893a293137d3ba2f2ab6c14f5062371c9a886d4952abd3beb853e3e6ba4e"
Nov 03 01:30:26.301011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.300985    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbbd\" (UniqueName: \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.301124 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.301105    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.301289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.301272    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.301337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.301293    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wn7\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-kube-api-access-s5wn7\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.301337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.301304    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.301337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.301313    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.402363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.402329    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbbd\" (UniqueName: \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.402543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.402429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.403104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.403079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.410932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.410910    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbbd\" (UniqueName: \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\") pod \"payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.573077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.573039    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:26.945569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:26.945535    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:30:26.946416 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:26.946387    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb053be2_0283_41dc_aab5_8bc980205edb.slice/crio-80245403c0b3cf43fbf80908e2ed10c2b9375d859d9d7c81be82aaaaf8524b1b WatchSource:0}: Error finding container 80245403c0b3cf43fbf80908e2ed10c2b9375d859d9d7c81be82aaaaf8524b1b: Status 404 returned error can't find the container with id 80245403c0b3cf43fbf80908e2ed10c2b9375d859d9d7c81be82aaaaf8524b1b
Nov 03 01:30:27.073996 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.073965    4531 scope.go:122] "RemoveContainer" containerID="c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"
Nov 03 01:30:27.074497 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.074464    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f_odh-ai-gateway-infra(408dcbcb-fb13-4ccf-a936-8d486fcc11f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5"
Nov 03 01:30:27.076028 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.076002    4531 scope.go:122] "RemoveContainer" containerID="24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"
Nov 03 01:30:27.076417 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.076374    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v_odh-ai-gateway-infra(3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8"
Nov 03 01:30:27.077848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.077827    4531 scope.go:122] "RemoveContainer" containerID="4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"
Nov 03 01:30:27.078199 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.078180    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg_odh-ai-gateway-infra(b3f6310c-f0ea-44c2-8109-7ba8d241b3f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1"
Nov 03 01:30:27.079628 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.079575    4531 scope.go:122] "RemoveContainer" containerID="f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"
Nov 03 01:30:27.080076 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.080048    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv_odh-ai-gateway-infra(2d50733f-a18e-43c7-9035-f0c331fd5511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511"
Nov 03 01:30:27.081750 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.081732    4531 scope.go:122] "RemoveContainer" containerID="b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"
Nov 03 01:30:27.082018 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.081995    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj_odh-ai-gateway-infra(934cfef1-1346-41b4-b41c-e0ec6b27aaee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee"
Nov 03 01:30:27.083348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.083327    4531 scope.go:122] "RemoveContainer" containerID="ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"
Nov 03 01:30:27.083601 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.083578    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp_odh-ai-gateway-infra(12b22bb9-80bf-460b-836e-2e8af3b30d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83"
Nov 03 01:30:27.085016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.084998    4531 scope.go:122] "RemoveContainer" containerID="0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"
Nov 03 01:30:27.085320 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.085296    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm_odh-ai-gateway-infra(26b335e2-51de-4e95-85f2-75b256c50838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" podUID="26b335e2-51de-4e95-85f2-75b256c50838"
Nov 03 01:30:27.086276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.086249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl" event={"ID":"bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf","Type":"ContainerDied","Data":"cb0b5ce73a254b5b067605d105e2468ba855344d7bab35c5f85d0c4e2e59d813"}
Nov 03 01:30:27.086505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.086483    4531 scope.go:122] "RemoveContainer" containerID="fb1d79abe62b243d142f19787536bda2f0057b09ff3ec6e4540f2518b2e07071"
Nov 03 01:30:27.086609 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.086261    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"
Nov 03 01:30:27.089774 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.089751    4531 scope.go:122] "RemoveContainer" containerID="aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"
Nov 03 01:30:27.090046 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.090021    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw_odh-ai-gateway-infra(04b00673-3454-4536-bc2b-571820696dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" podUID="04b00673-3454-4536-bc2b-571820696dbc"
Nov 03 01:30:27.092361 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.092336    4531 scope.go:122] "RemoveContainer" containerID="5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"
Nov 03 01:30:27.092617 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.092584    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8_odh-ai-gateway-infra(e4e12344-17f1-4dc7-bd43-176d0f99b2c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9"
Nov 03 01:30:27.094752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.094734    4531 scope.go:122] "RemoveContainer" containerID="63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"
Nov 03 01:30:27.094970 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:27.094950    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml_odh-ai-gateway-infra(3d4447f1-1a4d-4645-88f4-a424a6d94bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3"
Nov 03 01:30:27.096304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.096279    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" event={"ID":"eb053be2-0283-41dc-aab5-8bc980205edb","Type":"ContainerStarted","Data":"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"}
Nov 03 01:30:27.096447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.096432    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:27.096555 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.096540    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" event={"ID":"eb053be2-0283-41dc-aab5-8bc980205edb","Type":"ContainerStarted","Data":"80245403c0b3cf43fbf80908e2ed10c2b9375d859d9d7c81be82aaaaf8524b1b"}
Nov 03 01:30:27.132473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.131904    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:30:27.140199 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.140149    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-d6gzl"]
Nov 03 01:30:27.281418 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:27.281322    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" podStartSLOduration=65.28130827 podStartE2EDuration="1m5.28130827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:27.276196707 +0000 UTC m=+2972.489940368" watchObservedRunningTime="2026-11-03 01:30:27.28130827 +0000 UTC m=+2972.495051943"
Nov 03 01:30:28.906066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:28.906032    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf" path="/var/lib/kubelet/pods/bcf9ec75-9c0b-4c2e-a85b-4901e3ffd4bf/volumes"
Nov 03 01:30:33.896230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:33.896148    4531 scope.go:122] "RemoveContainer" containerID="d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"
Nov 03 01:30:34.910791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:34.910764    4531 scope.go:122] "RemoveContainer" containerID="a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"
Nov 03 01:30:34.911338 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:34.911008    4531 scope.go:122] "RemoveContainer" containerID="cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"
Nov 03 01:30:35.194070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.193980    4531 generic.go:356] "Generic (PLEG): container finished" podID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerID="66f3494d37ef8ca9bbe30361a40b70e5449507940ff9db11332cf1530407d36e" exitCode=60
Nov 03 01:30:35.194070 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.194051    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerDied","Data":"66f3494d37ef8ca9bbe30361a40b70e5449507940ff9db11332cf1530407d36e"}
Nov 03 01:30:35.194273 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.194142    4531 scope.go:122] "RemoveContainer" containerID="d6b8f3d8193548514de05123adacc1bef913bbcd10bf775e32dfec3a49dae35e"
Nov 03 01:30:35.194614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.194595    4531 scope.go:122] "RemoveContainer" containerID="66f3494d37ef8ca9bbe30361a40b70e5449507940ff9db11332cf1530407d36e"
Nov 03 01:30:35.195112 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:35.195068    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6_odh-ai-gateway-infra(64550299-3851-43d1-b2ac-4a2ddb450ac7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7"
Nov 03 01:30:35.895836 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.895795    4531 scope.go:122] "RemoveContainer" containerID="f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"
Nov 03 01:30:35.896150 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.896129    4531 scope.go:122] "RemoveContainer" containerID="b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"
Nov 03 01:30:35.896304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:35.896228    4531 scope.go:122] "RemoveContainer" containerID="63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"
Nov 03 01:30:36.205154 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.205121    4531 generic.go:356] "Generic (PLEG): container finished" podID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerID="eeedd51451d8fc03321523eb8195fc288a6d15c376148daa5dcd38437139b8d4" exitCode=60
Nov 03 01:30:36.205641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.205199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerDied","Data":"eeedd51451d8fc03321523eb8195fc288a6d15c376148daa5dcd38437139b8d4"}
Nov 03 01:30:36.205641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.205269    4531 scope.go:122] "RemoveContainer" containerID="a5bcb435ec82989e43d5868c9d1c622bc8f1333c39ea0de77bb410a592576264"
Nov 03 01:30:36.205776 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.205756    4531 scope.go:122] "RemoveContainer" containerID="eeedd51451d8fc03321523eb8195fc288a6d15c376148daa5dcd38437139b8d4"
Nov 03 01:30:36.206354 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:36.206279    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm_odh-ai-gateway-infra(b807a829-b4a3-4df2-ac05-9bf0601b4e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11"
Nov 03 01:30:36.207541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.207520    4531 generic.go:356] "Generic (PLEG): container finished" podID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerID="e7f0d46a7cb4197bda5a8828df2315c5311b8761c0fa7db56abbaf3352915bcc" exitCode=60
Nov 03 01:30:36.207632 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.207596    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerDied","Data":"e7f0d46a7cb4197bda5a8828df2315c5311b8761c0fa7db56abbaf3352915bcc"}
Nov 03 01:30:36.207988 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.207973    4531 scope.go:122] "RemoveContainer" containerID="e7f0d46a7cb4197bda5a8828df2315c5311b8761c0fa7db56abbaf3352915bcc"
Nov 03 01:30:36.208346 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:36.208322    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv_odh-ai-gateway-infra(dc1063ff-cc0e-43fd-ace6-348f52a17941)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941"
Nov 03 01:30:36.224086 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.224063    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:36.274943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.274918    4531 scope.go:122] "RemoveContainer" containerID="cdc00c5bcb59566bf879037036b548d662eab9538a4c4bbb9b04ca4207b4b902"
Nov 03 01:30:36.465701 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.465672    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:36.599180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.597871    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\") pod \"64550299-3851-43d1-b2ac-4a2ddb450ac7\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") "
Nov 03 01:30:36.599180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.598033    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\" (UniqueName: \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\") pod \"64550299-3851-43d1-b2ac-4a2ddb450ac7\" (UID: \"64550299-3851-43d1-b2ac-4a2ddb450ac7\") "
Nov 03 01:30:36.599717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.599693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca" pod "64550299-3851-43d1-b2ac-4a2ddb450ac7" (UID: "64550299-3851-43d1-b2ac-4a2ddb450ac7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:36.606902 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.606868    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4" pod "64550299-3851-43d1-b2ac-4a2ddb450ac7" (UID: "64550299-3851-43d1-b2ac-4a2ddb450ac7"). InnerVolumeSpecName "kube-api-access-wzfz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:36.700394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.700300    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfz4\" (UniqueName: \"kubernetes.io/projected/64550299-3851-43d1-b2ac-4a2ddb450ac7-kube-api-access-wzfz4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:36.700394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.700337    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64550299-3851-43d1-b2ac-4a2ddb450ac7-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:36.913743 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.913712    4531 scope.go:122] "RemoveContainer" containerID="c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"
Nov 03 01:30:36.915012 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:36.914985    4531 scope.go:122] "RemoveContainer" containerID="c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"
Nov 03 01:30:37.269881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.263558    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerID="0503a5475f20d6d658da19633e30445194073a0eb4a34d506861c88559ce2982" exitCode=60
Nov 03 01:30:37.270615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.270571    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerDied","Data":"0503a5475f20d6d658da19633e30445194073a0eb4a34d506861c88559ce2982"}
Nov 03 01:30:37.270839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.270818    4531 scope.go:122] "RemoveContainer" containerID="f007093bc8eaa7a9c26a1f6bb12f882b01c800d8f8ac8fa903da7dd109b29850"
Nov 03 01:30:37.280249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.278206    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:37.282658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.282637    4531 scope.go:122] "RemoveContainer" containerID="0503a5475f20d6d658da19633e30445194073a0eb4a34d506861c88559ce2982"
Nov 03 01:30:37.283106 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:37.283077    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv_odh-ai-gateway-infra(2d50733f-a18e-43c7-9035-f0c331fd5511)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511"
Nov 03 01:30:37.293137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.293111    4531 generic.go:356] "Generic (PLEG): container finished" podID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerID="61cb6a270bf187123a42039bfe3aaf9c8487c4cad01fcdbdd0d9b0c8af1cee2b" exitCode=60
Nov 03 01:30:37.293367 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.293198    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerDied","Data":"61cb6a270bf187123a42039bfe3aaf9c8487c4cad01fcdbdd0d9b0c8af1cee2b"}
Nov 03 01:30:37.293699 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.293685    4531 scope.go:122] "RemoveContainer" containerID="61cb6a270bf187123a42039bfe3aaf9c8487c4cad01fcdbdd0d9b0c8af1cee2b"
Nov 03 01:30:37.294096 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:37.294072    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj_odh-ai-gateway-infra(934cfef1-1346-41b4-b41c-e0ec6b27aaee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee"
Nov 03 01:30:37.297251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.297077    4531 generic.go:356] "Generic (PLEG): container finished" podID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerID="764e35fd1c53695e8873efeef6b29390194d99f17b68ded1613af575781f255b" exitCode=60
Nov 03 01:30:37.298237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.297422    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerDied","Data":"764e35fd1c53695e8873efeef6b29390194d99f17b68ded1613af575781f255b"}
Nov 03 01:30:37.298237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.297902    4531 scope.go:122] "RemoveContainer" containerID="764e35fd1c53695e8873efeef6b29390194d99f17b68ded1613af575781f255b"
Nov 03 01:30:37.299079 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:37.298401    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml_odh-ai-gateway-infra(3d4447f1-1a4d-4645-88f4-a424a6d94bc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3"
Nov 03 01:30:37.299862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.299834    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6" event={"ID":"64550299-3851-43d1-b2ac-4a2ddb450ac7","Type":"ContainerDied","Data":"443a67a94a79fc71f1e04dc5e96d547946a8b4855b60c9f028f6598189b2bab1"}
Nov 03 01:30:37.300012 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.299925    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"
Nov 03 01:30:37.303439 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.303231    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:37.361560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.361524    4531 scope.go:122] "RemoveContainer" containerID="b39daf8ba4d47c9f3c59eec5f83d9ff16c1257d5930556d2801af23c2c2d3666"
Nov 03 01:30:37.414366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.414333    4531 scope.go:122] "RemoveContainer" containerID="63426210366bb6630ce289d1f66aaae80617d828ec714b67f5e7af572d8b1098"
Nov 03 01:30:37.423839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.423772    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:37.435600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.435516    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-88bbf7-29894490-zfvx6"]
Nov 03 01:30:37.462451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.462431    4531 scope.go:122] "RemoveContainer" containerID="66f3494d37ef8ca9bbe30361a40b70e5449507940ff9db11332cf1530407d36e"
Nov 03 01:30:37.576673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.576544    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:37.636714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.636368    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:37.723915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.723877    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\") pod \"dc1063ff-cc0e-43fd-ace6-348f52a17941\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") "
Nov 03 01:30:37.724097 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.723969    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\") pod \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") "
Nov 03 01:30:37.724304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.724273    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\" (UniqueName: \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\") pod \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\" (UID: \"b807a829-b4a3-4df2-ac05-9bf0601b4e11\") "
Nov 03 01:30:37.724455 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.724374    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\" (UniqueName: \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\") pod \"dc1063ff-cc0e-43fd-ace6-348f52a17941\" (UID: \"dc1063ff-cc0e-43fd-ace6-348f52a17941\") "
Nov 03 01:30:37.724579 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.724551    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca" pod "dc1063ff-cc0e-43fd-ace6-348f52a17941" (UID: "dc1063ff-cc0e-43fd-ace6-348f52a17941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:37.724673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.724648    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca" pod "b807a829-b4a3-4df2-ac05-9bf0601b4e11" (UID: "b807a829-b4a3-4df2-ac05-9bf0601b4e11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:37.728062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.725368    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1063ff-cc0e-43fd-ace6-348f52a17941-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:37.728062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.725396    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b807a829-b4a3-4df2-ac05-9bf0601b4e11-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:37.739863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.739781    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p" pod "b807a829-b4a3-4df2-ac05-9bf0601b4e11" (UID: "b807a829-b4a3-4df2-ac05-9bf0601b4e11"). InnerVolumeSpecName "kube-api-access-wpc7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:37.742033 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.742000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw" pod "dc1063ff-cc0e-43fd-ace6-348f52a17941" (UID: "dc1063ff-cc0e-43fd-ace6-348f52a17941"). InnerVolumeSpecName "kube-api-access-89ksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:37.827913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.827871    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpc7p\" (UniqueName: \"kubernetes.io/projected/b807a829-b4a3-4df2-ac05-9bf0601b4e11-kube-api-access-wpc7p\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:37.827913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.827912    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89ksw\" (UniqueName: \"kubernetes.io/projected/dc1063ff-cc0e-43fd-ace6-348f52a17941-kube-api-access-89ksw\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:37.897013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.896979    4531 scope.go:122] "RemoveContainer" containerID="24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"
Nov 03 01:30:37.897271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:37.897243    4531 scope.go:122] "RemoveContainer" containerID="ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"
Nov 03 01:30:38.110687 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.110651    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:30:38.314810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.314775    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerStarted","Data":"903927d80f2fc066d07d8d5dd609d78b077ca13e55cc3c2916bfd652e21439da"}
Nov 03 01:30:38.318682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.318649    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm" event={"ID":"b807a829-b4a3-4df2-ac05-9bf0601b4e11","Type":"ContainerDied","Data":"c4644606814e6b0fc5077ec4f6c9d4b94f9df025ee25428d2559bb169b5f0f99"}
Nov 03 01:30:38.318829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.318716    4531 scope.go:122] "RemoveContainer" containerID="eeedd51451d8fc03321523eb8195fc288a6d15c376148daa5dcd38437139b8d4"
Nov 03 01:30:38.318829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.318810    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"
Nov 03 01:30:38.339279 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.339256    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"
Nov 03 01:30:38.340064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.339231    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv" event={"ID":"dc1063ff-cc0e-43fd-ace6-348f52a17941","Type":"ContainerDied","Data":"5136b9ccd41bbdc869dea142a214ac1bf522f8e45eca40ce68d4e11c8476eb8f"}
Nov 03 01:30:38.343524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.342136    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:38.347854 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.347824    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerStarted","Data":"83bcfb2a448631c009db22af65bc7ca75fb15b89f0330dc90cb133645058fbe6"}
Nov 03 01:30:38.388129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.388092    4531 scope.go:122] "RemoveContainer" containerID="e7f0d46a7cb4197bda5a8828df2315c5311b8761c0fa7db56abbaf3352915bcc"
Nov 03 01:30:38.409834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.407323    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:38.423216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.423140    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:38.679872 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.679793    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:38.689036 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.688965    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3c5983-29894490-xsqqm"]
Nov 03 01:30:38.706618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.706557    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:38.726284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.726213    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-57079b-29894490-qhqhv"]
Nov 03 01:30:38.923606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.921976    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" path="/var/lib/kubelet/pods/64550299-3851-43d1-b2ac-4a2ddb450ac7/volumes"
Nov 03 01:30:38.923606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.922549    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" path="/var/lib/kubelet/pods/b807a829-b4a3-4df2-ac05-9bf0601b4e11/volumes"
Nov 03 01:30:38.923606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.923142    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" path="/var/lib/kubelet/pods/dc1063ff-cc0e-43fd-ace6-348f52a17941/volumes"
Nov 03 01:30:38.925484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.924638    4531 scope.go:122] "RemoveContainer" containerID="4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"
Nov 03 01:30:38.925484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.924755    4531 scope.go:122] "RemoveContainer" containerID="5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"
Nov 03 01:30:38.925484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:38.925230    4531 scope.go:122] "RemoveContainer" containerID="aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"
Nov 03 01:30:39.021831 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.021465    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:39.113545 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.113520    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:39.117869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.117840    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.152083    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153062    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153087    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153106    4531 cpu_manager.go:414] "containerMap: removing container" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153114    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153125    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153132    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153153    4531 cpu_manager.go:414] "containerMap: removing container" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153178    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153192    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153201    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153211    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153219    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153246    4531 cpu_manager.go:414] "containerMap: removing container" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153253    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153262    4531 cpu_manager.go:414] "containerMap: removing container" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153269    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153287    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153295    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153311    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153318    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153334    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153342    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153366    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153372    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153383    4531 cpu_manager.go:414] "containerMap: removing container" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153391    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153407    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153414    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153427    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153433    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153593    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153603    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153617    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153626    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153634    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153645    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153654    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153663    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153671    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153680    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153692    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153702    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1063ff-cc0e-43fd-ace6-348f52a17941" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153711    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:30:39.155665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.153721    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:30:39.165248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.164010    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:30:39.165248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.164106    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\" (UniqueName: \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\") pod \"2d50733f-a18e-43c7-9035-f0c331fd5511\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") "
Nov 03 01:30:39.165248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.164128    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.165248 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.164293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\") pod \"2d50733f-a18e-43c7-9035-f0c331fd5511\" (UID: \"2d50733f-a18e-43c7-9035-f0c331fd5511\") "
Nov 03 01:30:39.165573 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.165371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca" pod "2d50733f-a18e-43c7-9035-f0c331fd5511" (UID: "2d50733f-a18e-43c7-9035-f0c331fd5511"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:39.171138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.170765    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5dab6a-serving-cert\""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.264828    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6" pod "2d50733f-a18e-43c7-9035-f0c331fd5511" (UID: "2d50733f-a18e-43c7-9035-f0c331fd5511"). InnerVolumeSpecName "kube-api-access-fmsh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.267549    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\" (UniqueName: \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\") pod \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") "
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.267749    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\") pod \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") "
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.267882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\") pod \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\" (UID: \"934cfef1-1346-41b4-b41c-e0ec6b27aaee\") "
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.268090    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\" (UniqueName: \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\") pod \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\" (UID: \"3d4447f1-1a4d-4645-88f4-a424a6d94bc3\") "
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.268843    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.268947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.268981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhtq\" (UniqueName: \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.278278    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca" pod "934cfef1-1346-41b4-b41c-e0ec6b27aaee" (UID: "934cfef1-1346-41b4-b41c-e0ec6b27aaee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.278970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.279273    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmsh6\" (UniqueName: \"kubernetes.io/projected/2d50733f-a18e-43c7-9035-f0c331fd5511-kube-api-access-fmsh6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.279296    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d50733f-a18e-43c7-9035-f0c331fd5511-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.279327    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934cfef1-1346-41b4-b41c-e0ec6b27aaee-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.290679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v" pod "934cfef1-1346-41b4-b41c-e0ec6b27aaee" (UID: "934cfef1-1346-41b4-b41c-e0ec6b27aaee"). InnerVolumeSpecName "kube-api-access-m7v4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:39.296047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.291341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca" pod "3d4447f1-1a4d-4645-88f4-a424a6d94bc3" (UID: "3d4447f1-1a4d-4645-88f4-a424a6d94bc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:39.332599 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.332559    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb" pod "3d4447f1-1a4d-4645-88f4-a424a6d94bc3" (UID: "3d4447f1-1a4d-4645-88f4-a424a6d94bc3"). InnerVolumeSpecName "kube-api-access-d8wfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:39.374967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.374934    4531 generic.go:356] "Generic (PLEG): container finished" podID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerID="83bcfb2a448631c009db22af65bc7ca75fb15b89f0330dc90cb133645058fbe6" exitCode=60
Nov 03 01:30:39.375278 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.375006    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerDied","Data":"83bcfb2a448631c009db22af65bc7ca75fb15b89f0330dc90cb133645058fbe6"}
Nov 03 01:30:39.375413 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.375363    4531 scope.go:122] "RemoveContainer" containerID="c691669aeb041108e7485ad76d5c695b63b757c511b6cb7f850675af6e19a948"
Nov 03 01:30:39.375995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.375972    4531 scope.go:122] "RemoveContainer" containerID="83bcfb2a448631c009db22af65bc7ca75fb15b89f0330dc90cb133645058fbe6"
Nov 03 01:30:39.376599 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:39.376570    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9_odh-ai-gateway-infra(76f53f02-45ad-44bd-bf1e-f519f5332df2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2"
Nov 03 01:30:39.378120 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.378094    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv" event={"ID":"2d50733f-a18e-43c7-9035-f0c331fd5511","Type":"ContainerDied","Data":"129b6dca9572eec9e80c917419a3a5016fb20be328a99f2327fcd344e65e282f"}
Nov 03 01:30:39.378308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.378147    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"
Nov 03 01:30:39.379870 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.379820    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj" event={"ID":"934cfef1-1346-41b4-b41c-e0ec6b27aaee","Type":"ContainerDied","Data":"f429e3d2f654950d4b3fe201558eed1d0ab82a03a045596fb35838bf31cabcb9"}
Nov 03 01:30:39.379962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.379908    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"
Nov 03 01:30:39.380659 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.380515    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.380659 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.380624    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.380821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.380685    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.380821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.380711    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhtq\" (UniqueName: \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.381844 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.381821    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8wfb\" (UniqueName: \"kubernetes.io/projected/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-kube-api-access-d8wfb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.382617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.382589    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.383465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.383443    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7v4v\" (UniqueName: \"kubernetes.io/projected/934cfef1-1346-41b4-b41c-e0ec6b27aaee-kube-api-access-m7v4v\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.383569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.383474    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4447f1-1a4d-4645-88f4-a424a6d94bc3-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.383952 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.383929    4531 generic.go:356] "Generic (PLEG): container finished" podID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerID="381268b311123da03e4dce5fac23cd7deaa60320bfda98a3a8dd360292202b83" exitCode=60
Nov 03 01:30:39.384114 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.384095    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerDied","Data":"381268b311123da03e4dce5fac23cd7deaa60320bfda98a3a8dd360292202b83"}
Nov 03 01:30:39.384498 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.384471    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.384648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.384559    4531 scope.go:122] "RemoveContainer" containerID="381268b311123da03e4dce5fac23cd7deaa60320bfda98a3a8dd360292202b83"
Nov 03 01:30:39.385099 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:39.385064    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp_odh-ai-gateway-infra(12b22bb9-80bf-460b-836e-2e8af3b30d83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83"
Nov 03 01:30:39.387262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.387238    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.388262 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.388238    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml" event={"ID":"3d4447f1-1a4d-4645-88f4-a424a6d94bc3","Type":"ContainerDied","Data":"a0ded55a9af8093bb1086b3f6b84ab00c73c573c47a1c327739989b9db7f3b0f"}
Nov 03 01:30:39.388364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.388351    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"
Nov 03 01:30:39.392755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.392728    4531 generic.go:356] "Generic (PLEG): container finished" podID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerID="903927d80f2fc066d07d8d5dd609d78b077ca13e55cc3c2916bfd652e21439da" exitCode=60
Nov 03 01:30:39.392901 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.392812    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerDied","Data":"903927d80f2fc066d07d8d5dd609d78b077ca13e55cc3c2916bfd652e21439da"}
Nov 03 01:30:39.394210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.393223    4531 scope.go:122] "RemoveContainer" containerID="903927d80f2fc066d07d8d5dd609d78b077ca13e55cc3c2916bfd652e21439da"
Nov 03 01:30:39.394210 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:39.393526    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f_odh-ai-gateway-infra(408dcbcb-fb13-4ccf-a936-8d486fcc11f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5"
Nov 03 01:30:39.397852 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.397833    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:39.399594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.399572    4531 generic.go:356] "Generic (PLEG): container finished" podID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerID="a657e36b7cefeb691a99c96b124ff3a0e99a39e6d9b5fe9068c16f8cd8c228de" exitCode=60
Nov 03 01:30:39.399713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.399645    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerDied","Data":"a657e36b7cefeb691a99c96b124ff3a0e99a39e6d9b5fe9068c16f8cd8c228de"}
Nov 03 01:30:39.400126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.400110    4531 scope.go:122] "RemoveContainer" containerID="a657e36b7cefeb691a99c96b124ff3a0e99a39e6d9b5fe9068c16f8cd8c228de"
Nov 03 01:30:39.400492 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:39.400471    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v_odh-ai-gateway-infra(3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8"
Nov 03 01:30:39.403752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.403491    4531 scope.go:122] "RemoveContainer" containerID="0503a5475f20d6d658da19633e30445194073a0eb4a34d506861c88559ce2982"
Nov 03 01:30:39.404204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.404176    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhtq\" (UniqueName: \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\") pod \"maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.419693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.419662    4531 scope.go:122] "RemoveContainer" containerID="61cb6a270bf187123a42039bfe3aaf9c8487c4cad01fcdbdd0d9b0c8af1cee2b"
Nov 03 01:30:39.437619 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.437595    4531 scope.go:122] "RemoveContainer" containerID="ebc03e10ea6a9fd9c3a2edd9178c6fd664e32b612c94bbcdd630490c342b561c"
Nov 03 01:30:39.453720 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.453029    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:39.456999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.456972    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4a697f-29894490-8qhml"]
Nov 03 01:30:39.460560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.460538    4531 scope.go:122] "RemoveContainer" containerID="764e35fd1c53695e8873efeef6b29390194d99f17b68ded1613af575781f255b"
Nov 03 01:30:39.469615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.469593    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:39.477541 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.477513    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef434a-29894490-qzpxj"]
Nov 03 01:30:39.488279 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.488255    4531 scope.go:122] "RemoveContainer" containerID="c778309b670f4e6e70f806e120ad8aeb895e5cee11b5746ad1dd17c0e3fde437"
Nov 03 01:30:39.498109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.498083    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:39.504319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.504293    4531 scope.go:122] "RemoveContainer" containerID="24ef2180b07096b7bc48843b79e38f950ad2072606d504f6de0392d31d8870c0"
Nov 03 01:30:39.505415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.505342    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:39.512611 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.512590    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5bfc51-29894490-gs8kv"]
Nov 03 01:30:39.561756 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.561729    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:39.849958 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:30:39.849923    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9682bde_7848_468a_9720_777f57154a68.slice/crio-8bb6bce8f0bff013ac1529e930a47c5c4497d7cfe74a9b400368229f61d4031a WatchSource:0}: Error finding container 8bb6bce8f0bff013ac1529e930a47c5c4497d7cfe74a9b400368229f61d4031a: Status 404 returned error can't find the container with id 8bb6bce8f0bff013ac1529e930a47c5c4497d7cfe74a9b400368229f61d4031a
Nov 03 01:30:39.850085 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.850064    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:30:39.883962 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:39.883928    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:30:39.895916 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:39.895888    4531 scope.go:122] "RemoveContainer" containerID="0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"
Nov 03 01:30:40.418917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.418881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" event={"ID":"a9682bde-7848-468a-9720-777f57154a68","Type":"ContainerStarted","Data":"8bb6bce8f0bff013ac1529e930a47c5c4497d7cfe74a9b400368229f61d4031a"}
Nov 03 01:30:40.421047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.421021    4531 generic.go:356] "Generic (PLEG): container finished" podID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerID="b589802afdeaa5e1681874538d3502f5ab1448b22d9bde784390ebf648a519a4" exitCode=60
Nov 03 01:30:40.421117 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.421081    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerDied","Data":"b589802afdeaa5e1681874538d3502f5ab1448b22d9bde784390ebf648a519a4"}
Nov 03 01:30:40.421219 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.421145    4531 scope.go:122] "RemoveContainer" containerID="4be6e8d86e6f14f653c4843e7aa5f138e7ae552eef04b2ae2ab6655fd6c804db"
Nov 03 01:30:40.421658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.421639    4531 scope.go:122] "RemoveContainer" containerID="b589802afdeaa5e1681874538d3502f5ab1448b22d9bde784390ebf648a519a4"
Nov 03 01:30:40.422011 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:40.421979    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg_odh-ai-gateway-infra(b3f6310c-f0ea-44c2-8109-7ba8d241b3f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1"
Nov 03 01:30:40.436531 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.434933    4531 generic.go:356] "Generic (PLEG): container finished" podID="04b00673-3454-4536-bc2b-571820696dbc" containerID="ef6a9ca0c0c7e01ce0f503c2d132cff9c5721b204977f56cf46813c7727a1868" exitCode=60
Nov 03 01:30:40.436531 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.435039    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerDied","Data":"ef6a9ca0c0c7e01ce0f503c2d132cff9c5721b204977f56cf46813c7727a1868"}
Nov 03 01:30:40.436531 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.435609    4531 scope.go:122] "RemoveContainer" containerID="ef6a9ca0c0c7e01ce0f503c2d132cff9c5721b204977f56cf46813c7727a1868"
Nov 03 01:30:40.437726 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:40.437352    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw_odh-ai-gateway-infra(04b00673-3454-4536-bc2b-571820696dbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" podUID="04b00673-3454-4536-bc2b-571820696dbc"
Nov 03 01:30:40.437857 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.437811    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:40.473387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.473364    4531 scope.go:122] "RemoveContainer" containerID="aced4733d1014b9935c057b32288ede7ad88d88edf3c916f414ae7e145dc9edc"
Nov 03 01:30:40.478424 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.478295    4531 generic.go:356] "Generic (PLEG): container finished" podID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerID="ad5429ff635e08e6fcf876bdb3280617c6523f6723501c7fec11b802ba9ec38a" exitCode=60
Nov 03 01:30:40.478424 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.478408    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerDied","Data":"ad5429ff635e08e6fcf876bdb3280617c6523f6723501c7fec11b802ba9ec38a"}
Nov 03 01:30:40.479133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.478917    4531 scope.go:122] "RemoveContainer" containerID="ad5429ff635e08e6fcf876bdb3280617c6523f6723501c7fec11b802ba9ec38a"
Nov 03 01:30:40.479442 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:40.479410    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8_odh-ai-gateway-infra(e4e12344-17f1-4dc7-bd43-176d0f99b2c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9"
Nov 03 01:30:40.568598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.519036    4531 scope.go:122] "RemoveContainer" containerID="5ec7ca83075a314ac7be689beb11e8aa2faa791436bc9552353e2b0cb1027126"
Nov 03 01:30:40.568598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.551859    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:40.660995 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.660642    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:40.819532 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.817662    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\" (UniqueName: \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\") pod \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") "
Nov 03 01:30:40.819532 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.817994    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\") pod \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\" (UID: \"408dcbcb-fb13-4ccf-a936-8d486fcc11f5\") "
Nov 03 01:30:40.820869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.820816    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca" pod "408dcbcb-fb13-4ccf-a936-8d486fcc11f5" (UID: "408dcbcb-fb13-4ccf-a936-8d486fcc11f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:40.822511 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.822475    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc" pod "408dcbcb-fb13-4ccf-a936-8d486fcc11f5" (UID: "408dcbcb-fb13-4ccf-a936-8d486fcc11f5"). InnerVolumeSpecName "kube-api-access-phqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:40.850015 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.849975    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:40.913228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.913190    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" path="/var/lib/kubelet/pods/2d50733f-a18e-43c7-9035-f0c331fd5511/volumes"
Nov 03 01:30:40.914367 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.914344    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" path="/var/lib/kubelet/pods/3d4447f1-1a4d-4645-88f4-a424a6d94bc3/volumes"
Nov 03 01:30:40.915295 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.915274    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" path="/var/lib/kubelet/pods/934cfef1-1346-41b4-b41c-e0ec6b27aaee/volumes"
Nov 03 01:30:40.926196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.924900    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\") pod \"76f53f02-45ad-44bd-bf1e-f519f5332df2\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") "
Nov 03 01:30:40.926196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.925103    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\" (UniqueName: \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\") pod \"76f53f02-45ad-44bd-bf1e-f519f5332df2\" (UID: \"76f53f02-45ad-44bd-bf1e-f519f5332df2\") "
Nov 03 01:30:40.926762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.926742    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:40.926927 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.926912    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phqnc\" (UniqueName: \"kubernetes.io/projected/408dcbcb-fb13-4ccf-a936-8d486fcc11f5-kube-api-access-phqnc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:40.930002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.929975    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca" pod "76f53f02-45ad-44bd-bf1e-f519f5332df2" (UID: "76f53f02-45ad-44bd-bf1e-f519f5332df2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:40.941975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:40.941946    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s" pod "76f53f02-45ad-44bd-bf1e-f519f5332df2" (UID: "76f53f02-45ad-44bd-bf1e-f519f5332df2"). InnerVolumeSpecName "kube-api-access-kdj8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:41.016728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.016699    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:41.029830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.029798    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f53f02-45ad-44bd-bf1e-f519f5332df2-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.029830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.029829    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj8s\" (UniqueName: \"kubernetes.io/projected/76f53f02-45ad-44bd-bf1e-f519f5332df2-kube-api-access-kdj8s\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.073560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.073492    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:41.131002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.130963    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\" (UniqueName: \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\") pod \"12b22bb9-80bf-460b-836e-2e8af3b30d83\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") "
Nov 03 01:30:41.131521 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.131494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\") pod \"12b22bb9-80bf-460b-836e-2e8af3b30d83\" (UID: \"12b22bb9-80bf-460b-836e-2e8af3b30d83\") "
Nov 03 01:30:41.134148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.134121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896" pod "12b22bb9-80bf-460b-836e-2e8af3b30d83" (UID: "12b22bb9-80bf-460b-836e-2e8af3b30d83"). InnerVolumeSpecName "kube-api-access-cj896". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:41.134942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.134917    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca" pod "12b22bb9-80bf-460b-836e-2e8af3b30d83" (UID: "12b22bb9-80bf-460b-836e-2e8af3b30d83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:41.232958 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.232924    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\") pod \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") "
Nov 03 01:30:41.233294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.233272    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\" (UniqueName: \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\") pod \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\" (UID: \"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8\") "
Nov 03 01:30:41.233392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.233370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca" pod "3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" (UID: "3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:41.234089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.234062    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj896\" (UniqueName: \"kubernetes.io/projected/12b22bb9-80bf-460b-836e-2e8af3b30d83-kube-api-access-cj896\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.234089 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.234085    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.234296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.234103    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b22bb9-80bf-460b-836e-2e8af3b30d83-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.235520 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.235486    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h" pod "3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" (UID: "3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8"). InnerVolumeSpecName "kube-api-access-9k55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:41.336139 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.336102    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k55h\" (UniqueName: \"kubernetes.io/projected/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8-kube-api-access-9k55h\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:41.492408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.492377    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"
Nov 03 01:30:41.492879 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.492423    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v" event={"ID":"3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8","Type":"ContainerDied","Data":"c4f8580216e5cded786146683eb5aa6004f9e9a686c066be14ba96fa9eabc046"}
Nov 03 01:30:41.492879 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.492488    4531 scope.go:122] "RemoveContainer" containerID="a657e36b7cefeb691a99c96b124ff3a0e99a39e6d9b5fe9068c16f8cd8c228de"
Nov 03 01:30:41.494507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.494482    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" event={"ID":"a9682bde-7848-468a-9720-777f57154a68","Type":"ContainerStarted","Data":"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"}
Nov 03 01:30:41.494634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.494563    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:41.498254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.498231    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"
Nov 03 01:30:41.498254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.498243    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9" event={"ID":"76f53f02-45ad-44bd-bf1e-f519f5332df2","Type":"ContainerDied","Data":"ccc8bc3267800093f71d00aa5b830c84ed0da94eda6b817f99ea3c936da6b352"}
Nov 03 01:30:41.499987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.499959    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"
Nov 03 01:30:41.500092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.500005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp" event={"ID":"12b22bb9-80bf-460b-836e-2e8af3b30d83","Type":"ContainerDied","Data":"824fb6cce7b7b27406eb311bcc09feca7676bd9cc5d9f654bf6a5184f0643311"}
Nov 03 01:30:41.508572 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.508549    4531 scope.go:122] "RemoveContainer" containerID="83bcfb2a448631c009db22af65bc7ca75fb15b89f0330dc90cb133645058fbe6"
Nov 03 01:30:41.508938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.508917    4531 generic.go:356] "Generic (PLEG): container finished" podID="26b335e2-51de-4e95-85f2-75b256c50838" containerID="4a906d909c3c22d5049d1ba42015bcbf8d4663a8cb4e72b1a2acf87ac8808641" exitCode=60
Nov 03 01:30:41.509060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.509015    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerDied","Data":"4a906d909c3c22d5049d1ba42015bcbf8d4663a8cb4e72b1a2acf87ac8808641"}
Nov 03 01:30:41.510603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.509944    4531 scope.go:122] "RemoveContainer" containerID="4a906d909c3c22d5049d1ba42015bcbf8d4663a8cb4e72b1a2acf87ac8808641"
Nov 03 01:30:41.510603 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:30:41.510320    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm_odh-ai-gateway-infra(26b335e2-51de-4e95-85f2-75b256c50838)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" podUID="26b335e2-51de-4e95-85f2-75b256c50838"
Nov 03 01:30:41.511055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.511029    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:41.635275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.549074    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f" event={"ID":"408dcbcb-fb13-4ccf-a936-8d486fcc11f5","Type":"ContainerDied","Data":"71e9909036b97851aedfb4b6f6a1961361df2b351e6b8034ece8ab067255d7c5"}
Nov 03 01:30:41.635275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.549276    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"
Nov 03 01:30:41.635275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.629975    4531 scope.go:122] "RemoveContainer" containerID="381268b311123da03e4dce5fac23cd7deaa60320bfda98a3a8dd360292202b83"
Nov 03 01:30:41.644708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.643829    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:41.654845 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.654779    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" podStartSLOduration=69.085297677 podStartE2EDuration="1m9.6547587s" totalImagesPullingTime="569.461023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:41.541784799 +0000 UTC m=+2986.755528483" watchObservedRunningTime="2026-11-03 01:30:41.6547587 +0000 UTC m=+2986.868502373"
Nov 03 01:30:41.716290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.709771    4531 scope.go:122] "RemoveContainer" containerID="0510f2991f9d2bba08eec1cc57631e435d49973e82bd79aecdccb5ee7da4f05d"
Nov 03 01:30:41.801381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.801354    4531 scope.go:122] "RemoveContainer" containerID="903927d80f2fc066d07d8d5dd609d78b077ca13e55cc3c2916bfd652e21439da"
Nov 03 01:30:41.802357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.802293    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:41.811130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.811104    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b6afd6-29894490-9ct2v"]
Nov 03 01:30:41.820364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.820336    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:41.830589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.830532    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:41.850198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.850136    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-650be0-29894490-5q7zp"]
Nov 03 01:30:41.875603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.875539    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:41.885392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.884818    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:41.886888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.886811    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-531447-29894490-2kqq9"]
Nov 03 01:30:41.898472 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.898425    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:41.909899 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.909870    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4af1f7-29894490-bfl6f"]
Nov 03 01:30:41.913903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.913869    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\" (UniqueName: \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\") pod \"04b00673-3454-4536-bc2b-571820696dbc\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") "
Nov 03 01:30:41.914129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.914109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\") pod \"04b00673-3454-4536-bc2b-571820696dbc\" (UID: \"04b00673-3454-4536-bc2b-571820696dbc\") "
Nov 03 01:30:41.915571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.915538    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca" pod "04b00673-3454-4536-bc2b-571820696dbc" (UID: "04b00673-3454-4536-bc2b-571820696dbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:41.921434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:41.921352    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9" pod "04b00673-3454-4536-bc2b-571820696dbc" (UID: "04b00673-3454-4536-bc2b-571820696dbc"). InnerVolumeSpecName "kube-api-access-vksd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:42.001700 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.000892    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:42.017123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.016906    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b00673-3454-4536-bc2b-571820696dbc-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.017123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.016934    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vksd9\" (UniqueName: \"kubernetes.io/projected/04b00673-3454-4536-bc2b-571820696dbc-kube-api-access-vksd9\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.118223 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.118186    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\" (UniqueName: \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\") pod \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") "
Nov 03 01:30:42.118461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.118438    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\") pod \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\" (UID: \"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1\") "
Nov 03 01:30:42.118829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.118805    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca" pod "b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" (UID: "b3f6310c-f0ea-44c2-8109-7ba8d241b3f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:42.120370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.119344    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.120867 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.120837    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv" pod "b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" (UID: "b3f6310c-f0ea-44c2-8109-7ba8d241b3f1"). InnerVolumeSpecName "kube-api-access-gq5qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:42.140690 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.140628    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:42.220672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.220635    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\" (UniqueName: \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\") pod \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") "
Nov 03 01:30:42.220946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.220929    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\") pod \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\" (UID: \"e4e12344-17f1-4dc7-bd43-176d0f99b2c9\") "
Nov 03 01:30:42.221405 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.221370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca" pod "e4e12344-17f1-4dc7-bd43-176d0f99b2c9" (UID: "e4e12344-17f1-4dc7-bd43-176d0f99b2c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:42.221762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.221738    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5qv\" (UniqueName: \"kubernetes.io/projected/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1-kube-api-access-gq5qv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.221856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.221767    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.223257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.223225    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4" pod "e4e12344-17f1-4dc7-bd43-176d0f99b2c9" (UID: "e4e12344-17f1-4dc7-bd43-176d0f99b2c9"). InnerVolumeSpecName "kube-api-access-l2wq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:42.323829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.323783    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wq4\" (UniqueName: \"kubernetes.io/projected/e4e12344-17f1-4dc7-bd43-176d0f99b2c9-kube-api-access-l2wq4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:42.562423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.562311    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"
Nov 03 01:30:42.562423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.562310    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg" event={"ID":"b3f6310c-f0ea-44c2-8109-7ba8d241b3f1","Type":"ContainerDied","Data":"1f29fdb2c0ec437c519d448b2e4b447b4a865284f588445269e004bd798b7a40"}
Nov 03 01:30:42.563503 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.562474    4531 scope.go:122] "RemoveContainer" containerID="b589802afdeaa5e1681874538d3502f5ab1448b22d9bde784390ebf648a519a4"
Nov 03 01:30:42.570410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.570389    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"
Nov 03 01:30:42.570614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.570386    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw" event={"ID":"04b00673-3454-4536-bc2b-571820696dbc","Type":"ContainerDied","Data":"7dfc2cdba31369219af4345c2e94ec45d258adea4b208bc07879fe025af98358"}
Nov 03 01:30:42.573601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.572442    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"
Nov 03 01:30:42.573601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.572515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8" event={"ID":"e4e12344-17f1-4dc7-bd43-176d0f99b2c9","Type":"ContainerDied","Data":"d86dd242cb98c2ca84cc11bab8341eaf6772fbb8ce3b69eb0672479cc9bfff0d"}
Nov 03 01:30:42.580373 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.580307    4531 scope.go:122] "RemoveContainer" containerID="ef6a9ca0c0c7e01ce0f503c2d132cff9c5721b204977f56cf46813c7727a1868"
Nov 03 01:30:42.594294 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.594267    4531 scope.go:122] "RemoveContainer" containerID="ad5429ff635e08e6fcf876bdb3280617c6523f6723501c7fec11b802ba9ec38a"
Nov 03 01:30:42.597551 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.597529    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:42.603727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.603697    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e1f08d-29894490-l6wbg"]
Nov 03 01:30:42.613597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.613574    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:42.622883 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.622858    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-020af9-29894490-6s5s8"]
Nov 03 01:30:42.634078 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.634051    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:42.651460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.651434    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ee618c-29894490-vvrdw"]
Nov 03 01:30:42.666600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.666567    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:42.873636 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.873607    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:42.924858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.924820    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b00673-3454-4536-bc2b-571820696dbc" path="/var/lib/kubelet/pods/04b00673-3454-4536-bc2b-571820696dbc/volumes"
Nov 03 01:30:42.925749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.925721    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" path="/var/lib/kubelet/pods/12b22bb9-80bf-460b-836e-2e8af3b30d83/volumes"
Nov 03 01:30:42.931387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.931361    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" path="/var/lib/kubelet/pods/3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8/volumes"
Nov 03 01:30:42.931998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.931979    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" path="/var/lib/kubelet/pods/408dcbcb-fb13-4ccf-a936-8d486fcc11f5/volumes"
Nov 03 01:30:42.932608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.932586    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" path="/var/lib/kubelet/pods/76f53f02-45ad-44bd-bf1e-f519f5332df2/volumes"
Nov 03 01:30:42.933686 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.933663    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" path="/var/lib/kubelet/pods/b3f6310c-f0ea-44c2-8109-7ba8d241b3f1/volumes"
Nov 03 01:30:42.934448 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.934424    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" path="/var/lib/kubelet/pods/e4e12344-17f1-4dc7-bd43-176d0f99b2c9/volumes"
Nov 03 01:30:42.938496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.938471    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\" (UniqueName: \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\") pod \"26b335e2-51de-4e95-85f2-75b256c50838\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") "
Nov 03 01:30:42.938602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.938567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\") pod \"26b335e2-51de-4e95-85f2-75b256c50838\" (UID: \"26b335e2-51de-4e95-85f2-75b256c50838\") "
Nov 03 01:30:42.939803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.939763    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca" pod "26b335e2-51de-4e95-85f2-75b256c50838" (UID: "26b335e2-51de-4e95-85f2-75b256c50838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:42.941482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:42.941449    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k" pod "26b335e2-51de-4e95-85f2-75b256c50838" (UID: "26b335e2-51de-4e95-85f2-75b256c50838"). InnerVolumeSpecName "kube-api-access-xjl8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:43.040950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.040908    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjl8k\" (UniqueName: \"kubernetes.io/projected/26b335e2-51de-4e95-85f2-75b256c50838-kube-api-access-xjl8k\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:43.040950 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.040941    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b335e2-51de-4e95-85f2-75b256c50838-maas-api-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:43.594799 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.594767    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"
Nov 03 01:30:43.595292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.594766    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm" event={"ID":"26b335e2-51de-4e95-85f2-75b256c50838","Type":"ContainerDied","Data":"9918c469526c98f36e7fa825bbf78e4be1957fad09f051b99e0d0f43036d84a5"}
Nov 03 01:30:43.595292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.594937    4531 scope.go:122] "RemoveContainer" containerID="4a906d909c3c22d5049d1ba42015bcbf8d4663a8cb4e72b1a2acf87ac8808641"
Nov 03 01:30:43.630037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.630002    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:43.635769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:43.635735    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-717217-29894490-vnxgm"]
Nov 03 01:30:44.907144 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:44.907102    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b335e2-51de-4e95-85f2-75b256c50838" path="/var/lib/kubelet/pods/26b335e2-51de-4e95-85f2-75b256c50838/volumes"
Nov 03 01:30:47.587734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:47.587696    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:30:56.819600 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:30:56.819567    4531 scope.go:122] "RemoveContainer" containerID="5fe75a69c505afebaaa30c370f6e62811d52fa3af6d25d2cb0170e1545fd3222"
Nov 03 01:31:09.880401 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:31:09.880311    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:31:18.863312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:18.863271    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 01:31:18.863928 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:18.863585    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy" containerID="cri-o://ce4ec99b201c5f4323b55fe021a6a2e7787690d6aea1310480509a90d230bb64" gracePeriod=30
Nov 03 01:31:24.171948 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.171910    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerID="ce4ec99b201c5f4323b55fe021a6a2e7787690d6aea1310480509a90d230bb64" exitCode=0
Nov 03 01:31:24.172359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.172071    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" event={"ID":"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a","Type":"ContainerDied","Data":"ce4ec99b201c5f4323b55fe021a6a2e7787690d6aea1310480509a90d230bb64"}
Nov 03 01:31:24.172359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.172110    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j" event={"ID":"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a","Type":"ContainerDied","Data":"58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd"}
Nov 03 01:31:24.172359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.172127    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d286db8d4de791437a38276fe00f02a23d4797957c081766270f68a50d54dd"
Nov 03 01:31:24.193221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.193187    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 01:31:24.288088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288052    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288100    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288203    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288241    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288274    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288348    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288378    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288460    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.288672 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.288514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\") pod \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\" (UID: \"9a344b50-a4c2-4d77-bfce-1756c0ba8d4a\") "
Nov 03 01:31:24.289242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.289101    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:24.289363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.289290    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:24.289467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.289425    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:24.289605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.289586    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:31:24.289703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.289678    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:24.292076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.292045    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:24.292237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.292107    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:24.298251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.298080    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:31:24.298251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.298211    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t" pod "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" (UID: "9a344b50-a4c2-4d77-bfce-1756c0ba8d4a"). InnerVolumeSpecName "kube-api-access-zkt5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393810    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393846    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393861    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393880    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkt5t\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-kube-api-access-zkt5t\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393897    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393909    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393925    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393938    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:24.394288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:24.393954    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:25.184805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:25.184716    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"
Nov 03 01:31:25.210076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:25.210032    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 01:31:25.215427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:25.215398    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-qrq8j"]
Nov 03 01:31:26.905568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:26.905536    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" path="/var/lib/kubelet/pods/9a344b50-a4c2-4d77-bfce-1756c0ba8d4a/volumes"
Nov 03 01:31:39.879487 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:31:39.879447    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:31:57.042569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:31:57.042531    4531 scope.go:122] "RemoveContainer" containerID="ce4ec99b201c5f4323b55fe021a6a2e7787690d6aea1310480509a90d230bb64"
Nov 03 01:32:09.881317 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:32:09.881276    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:32:39.880804 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:32:39.880720    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:33:09.886459 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:09.886414    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:33:32.192285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:32.191875    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:33:32.192285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:32.192125    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerName="maas-api" containerID="cri-o://776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95" gracePeriod=30
Nov 03 01:33:33.144500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.144456    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.149:8443/health\": dial tcp 10.133.0.149:8443: connect: connection refused"
Nov 03 01:33:33.539279 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.539243    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:33:33.577987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.577947    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\") pod \"7f384fe7-0d42-437e-b037-464d5da5ae86\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") "
Nov 03 01:33:33.578242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.578029    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\" (UniqueName: \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\") pod \"7f384fe7-0d42-437e-b037-464d5da5ae86\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") "
Nov 03 01:33:33.578242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.578063    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\") pod \"7f384fe7-0d42-437e-b037-464d5da5ae86\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") "
Nov 03 01:33:33.578504 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.578468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca" pod "7f384fe7-0d42-437e-b037-464d5da5ae86" (UID: "7f384fe7-0d42-437e-b037-464d5da5ae86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:33.578565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.578475    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\") pod \"7f384fe7-0d42-437e-b037-464d5da5ae86\" (UID: \"7f384fe7-0d42-437e-b037-464d5da5ae86\") "
Nov 03 01:33:33.579299 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.579270    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f384fe7-0d42-437e-b037-464d5da5ae86-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:33.583329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.583237    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs" pod "7f384fe7-0d42-437e-b037-464d5da5ae86" (UID: "7f384fe7-0d42-437e-b037-464d5da5ae86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:33.583329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.583304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls" pod "7f384fe7-0d42-437e-b037-464d5da5ae86" (UID: "7f384fe7-0d42-437e-b037-464d5da5ae86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:33.583329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.583299    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn" pod "7f384fe7-0d42-437e-b037-464d5da5ae86" (UID: "7f384fe7-0d42-437e-b037-464d5da5ae86"). InnerVolumeSpecName "kube-api-access-pd2xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:33.681825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.681781    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:33.681825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.681826    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2xn\" (UniqueName: \"kubernetes.io/projected/7f384fe7-0d42-437e-b037-464d5da5ae86-kube-api-access-pd2xn\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:33.682096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.681845    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f384fe7-0d42-437e-b037-464d5da5ae86-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:33.835451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.835418    4531 generic.go:356] "Generic (PLEG): container finished" podID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerID="776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95" exitCode=0
Nov 03 01:33:33.835668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.835473    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"
Nov 03 01:33:33.835668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.835505    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" event={"ID":"7f384fe7-0d42-437e-b037-464d5da5ae86","Type":"ContainerDied","Data":"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"}
Nov 03 01:33:33.835668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.835566    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc" event={"ID":"7f384fe7-0d42-437e-b037-464d5da5ae86","Type":"ContainerDied","Data":"0de83ab9904ec7158bac9a331d7d3f153223ab8d6b3c8c7a152b0fe1148787a2"}
Nov 03 01:33:33.835668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.835597    4531 scope.go:122] "RemoveContainer" containerID="776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"
Nov 03 01:33:33.852638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.852615    4531 scope.go:122] "RemoveContainer" containerID="776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"
Nov 03 01:33:33.852983 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:33.852963    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95\": container with ID starting with 776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95 not found: ID does not exist" containerID="776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"
Nov 03 01:33:33.853048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.852992    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95"} err="failed to get container status \"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95\": rpc error: code = NotFound desc = could not find container \"776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95\": container with ID starting with 776e9320eb01211dcf722ebd8b26e569ef1b004fe8ded4d6ff6854e2873a9d95 not found: ID does not exist"
Nov 03 01:33:33.864821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.864789    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:33:33.869341 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:33.869312    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee618c-75b48b85c8-zbrsc"]
Nov 03 01:33:34.907649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:34.907612    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" path="/var/lib/kubelet/pods/7f384fe7-0d42-437e-b037-464d5da5ae86/volumes"
Nov 03 01:33:37.002924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.001816    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:33:37.002924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.002820    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy" containerID="cri-o://6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191" gracePeriod=30
Nov 03 01:33:37.009405 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.009211    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:33:37.009557 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.009449    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" containerID="cri-o://bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d" gracePeriod=30
Nov 03 01:33:37.963676 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.963635    4531 patch_prober.go:29] interesting pod/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:33:37.963851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:37.963695    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:33:39.649188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.649136    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:33:39.649986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.649963    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.649986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.649981    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.649986 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.649991    4531 cpu_manager.go:414] "containerMap: removing container" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.649996    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650005    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650011    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650019    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650025    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650033    4531 cpu_manager.go:414] "containerMap: removing container" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650038    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650051    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650055    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650063    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650067    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650074    4531 cpu_manager.go:414] "containerMap: removing container" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650081    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650088    4531 cpu_manager.go:414] "containerMap: removing container" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650094    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934cfef1-1346-41b4-b41c-e0ec6b27aaee" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650101    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650105    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650111    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650115    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650127    4531 cpu_manager.go:414] "containerMap: removing container" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650132    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650137    4531 cpu_manager.go:414] "containerMap: removing container" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650141    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650148    4531 cpu_manager.go:414] "containerMap: removing container" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650152    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650172    4531 cpu_manager.go:414] "containerMap: removing container" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650178    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.650156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650188    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650194    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650210    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650217    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650225    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650230    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650237    4531 cpu_manager.go:414] "containerMap: removing container" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650243    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650258    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650265    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650282    4531 cpu_manager.go:414] "containerMap: removing container" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650288    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650301    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650307    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650319    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650326    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650338    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerName="maas-api"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650345    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerName="maas-api"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650549    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650562    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650569    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650577    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b807a829-b4a3-4df2-ac05-9bf0601b4e11" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650583    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650590    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650600    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650608    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a344b50-a4c2-4d77-bfce-1756c0ba8d4a" containerName="istio-proxy"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650616    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650623    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650629    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="64550299-3851-43d1-b2ac-4a2ddb450ac7" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650636    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650641    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650648    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650654    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f384fe7-0d42-437e-b037-464d5da5ae86" containerName="maas-api"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650661    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650670    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4447f1-1a4d-4645-88f4-a424a6d94bc3" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650679    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650686    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650693    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b00673-3454-4536-bc2b-571820696dbc" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650699    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650705    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650712    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650719    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650723    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:33:39.651421 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.650730    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d50733f-a18e-43c7-9035-f0c331fd5511" containerName="cleanup"
Nov 03 01:33:39.684220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.657618    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.684220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.666118    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-737bf2d8-serving-cert\""
Nov 03 01:33:39.684220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.671050    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:33:39.750898 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.750862    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6r8j\" (UniqueName: \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.750898 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.750906    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.751133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.751080    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.751395 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.751375    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.855221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.853102    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.855221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.853381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.855221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.853467    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6r8j\" (UniqueName: \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.855221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.853500    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.880194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.878711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.943561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.942313    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.943561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.943448    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.950543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.950478    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6r8j\" (UniqueName: \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\") pod \"maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:39.991521 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:39.991452    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:33:39.996067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:39.996036    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:40.335565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:40.331461    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:33:40.949324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:40.949284    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" event={"ID":"18ff3ca3-1d34-4469-ae60-1486f13d2f94","Type":"ContainerStarted","Data":"aa0c9307e238bf24b75584edd3b706db2991b600d33ae6a655070a76c755ac5b"}
Nov 03 01:33:41.959623 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:41.959584    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" event={"ID":"18ff3ca3-1d34-4469-ae60-1486f13d2f94","Type":"ContainerStarted","Data":"2416ac6595d493530865e457cc100c3328cecda896051645bda35d4fee474708"}
Nov 03 01:33:41.960013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:41.959674    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:41.977830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:41.977770    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" podStartSLOduration=2.371530459 podStartE2EDuration="2.977741596s" totalImagesPullingTime="606.211137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:41.977322725 +0000 UTC m=+3167.191066396" watchObservedRunningTime="2026-11-03 01:33:41.977741596 +0000 UTC m=+3167.191485269"
Nov 03 01:33:42.421582 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.421557    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:33:42.425129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.425107    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:33:42.534240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534066    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.534240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534185    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534240 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534223    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534252    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534333    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534498    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534546    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.534596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534589    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534752    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.534903    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.535281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.535005    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\") pod \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\" (UID: \"429ff11e-60e8-4d41-a6e3-c2981e342b1b\") "
Nov 03 01:33:42.535603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.535335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.535666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.535600    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.535803    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536019    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536152    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536299    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536498    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536698    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.536899    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537009    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\") pod \"6018817e-d13d-42cd-b420-3a15c387c78a\" (UID: \"6018817e-d13d-42cd-b420-3a15c387c78a\") "
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537869    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537893    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537907    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537923    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6018817e-d13d-42cd-b420-3a15c387c78a-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537939    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537956    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537971    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537984    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.537998    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.538011    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.539332    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.541381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.541437    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:42.543230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.543203    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:33:42.544449 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.544416    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:42.544830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.544788    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "kube-api-access-98zrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:42.545011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.544993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:42.545137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.545118    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5" pod "429ff11e-60e8-4d41-a6e3-c2981e342b1b" (UID: "429ff11e-60e8-4d41-a6e3-c2981e342b1b"). InnerVolumeSpecName "kube-api-access-ftxv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:42.546916 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.546881    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo" pod "6018817e-d13d-42cd-b420-3a15c387c78a" (UID: "6018817e-d13d-42cd-b420-3a15c387c78a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:33:42.640463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640420    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98zrw\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-kube-api-access-98zrw\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640464    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6018817e-d13d-42cd-b420-3a15c387c78a-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640483    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6018817e-d13d-42cd-b420-3a15c387c78a-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640500    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640514    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640528    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxv5\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-kube-api-access-ftxv5\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640542    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6018817e-d13d-42cd-b420-3a15c387c78a-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:42.640812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:42.640555    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429ff11e-60e8-4d41-a6e3-c2981e342b1b-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:43.024446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.024410    4531 generic.go:356] "Generic (PLEG): container finished" podID="6018817e-d13d-42cd-b420-3a15c387c78a" containerID="6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191" exitCode=0
Nov 03 01:33:43.024829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.024553    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"
Nov 03 01:33:43.025001 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.024970    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" event={"ID":"6018817e-d13d-42cd-b420-3a15c387c78a","Type":"ContainerDied","Data":"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"}
Nov 03 01:33:43.025110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.025068    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd" event={"ID":"6018817e-d13d-42cd-b420-3a15c387c78a","Type":"ContainerDied","Data":"cf10e36fe47497ab0691f91ea8b1dd5a618e011ae6157328395d55dfdf959696"}
Nov 03 01:33:43.025110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.025099    4531 scope.go:122] "RemoveContainer" containerID="6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"
Nov 03 01:33:43.027818 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.027790    4531 generic.go:356] "Generic (PLEG): container finished" podID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerID="bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d" exitCode=0
Nov 03 01:33:43.028671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.028633    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"
Nov 03 01:33:43.028930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.028906    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" event={"ID":"429ff11e-60e8-4d41-a6e3-c2981e342b1b","Type":"ContainerDied","Data":"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"}
Nov 03 01:33:43.029041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.028957    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7" event={"ID":"429ff11e-60e8-4d41-a6e3-c2981e342b1b","Type":"ContainerDied","Data":"e46d1174c8fe04615cc131dd3437b14c4a0cd7de28b799b467b960c201575b44"}
Nov 03 01:33:43.062922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.062901    4531 scope.go:122] "RemoveContainer" containerID="6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"
Nov 03 01:33:43.063326 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:43.063298    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191\": container with ID starting with 6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191 not found: ID does not exist" containerID="6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"
Nov 03 01:33:43.063407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.063339    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191"} err="failed to get container status \"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191\": rpc error: code = NotFound desc = could not find container \"6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191\": container with ID starting with 6cdbe56b88d17a840dcb56458546ffa42dad9d78a895c27339915fb8fa211191 not found: ID does not exist"
Nov 03 01:33:43.063407 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.063364    4531 scope.go:122] "RemoveContainer" containerID="bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"
Nov 03 01:33:43.075328 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.075298    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:33:43.082007 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.081982    4531 scope.go:122] "RemoveContainer" containerID="bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"
Nov 03 01:33:43.082087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.082069    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-zm4z7"]
Nov 03 01:33:43.082417 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:43.082395    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d\": container with ID starting with bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d not found: ID does not exist" containerID="bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"
Nov 03 01:33:43.082564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.082423    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d"} err="failed to get container status \"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d\": rpc error: code = NotFound desc = could not find container \"bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d\": container with ID starting with bd84104b3044da8487dbdc704c47c28165ad1ae65758bd5fc4d369b6810ace1d not found: ID does not exist"
Nov 03 01:33:43.092039 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.092013    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:33:43.096534 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:43.096510    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee618c-openshift-default-5ff9c74dc9-smljd"]
Nov 03 01:33:44.283599 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:44.283563    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:33:44.905423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:44.905393    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" path="/var/lib/kubelet/pods/429ff11e-60e8-4d41-a6e3-c2981e342b1b/volumes"
Nov 03 01:33:44.905874 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:44.905855    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" path="/var/lib/kubelet/pods/6018817e-d13d-42cd-b420-3a15c387c78a/volumes"
Nov 03 01:33:48.036067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:48.035981    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:33:49.284041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.283989    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" podUID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" containerName="payload-processing" containerID="cri-o://c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b" gracePeriod=25
Nov 03 01:33:49.642598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.642570    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:33:49.777411 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.777376    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\") pod \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") "
Nov 03 01:33:49.777613 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.777531    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\" (UniqueName: \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\") pod \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\" (UID: \"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd\") "
Nov 03 01:33:49.777822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.777797    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume" pod "adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" (UID: "adc1cf63-4645-4a69-ae98-5fa2dafa7fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:49.778148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.778133    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:49.779875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.779848    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf" pod "adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" (UID: "adc1cf63-4645-4a69-ae98-5fa2dafa7fbd"). InnerVolumeSpecName "kube-api-access-c6tcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:49.879395 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:49.879311    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tcf\" (UniqueName: \"kubernetes.io/projected/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd-kube-api-access-c6tcf\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:50.100233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.100189    4531 generic.go:356] "Generic (PLEG): container finished" podID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" containerID="c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b" exitCode=0
Nov 03 01:33:50.100426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.100264    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"
Nov 03 01:33:50.100426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.100263    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" event={"ID":"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd","Type":"ContainerDied","Data":"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"}
Nov 03 01:33:50.100426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.100403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll" event={"ID":"adc1cf63-4645-4a69-ae98-5fa2dafa7fbd","Type":"ContainerDied","Data":"f7170db3741ceff838e595efd45af41b2c0f0ca4b3efdef5d9d6eb94ad885dd2"}
Nov 03 01:33:50.100601 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.100435    4531 scope.go:122] "RemoveContainer" containerID="c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"
Nov 03 01:33:50.114803 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.114780    4531 scope.go:122] "RemoveContainer" containerID="c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"
Nov 03 01:33:50.115153 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:33:50.115127    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b\": container with ID starting with c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b not found: ID does not exist" containerID="c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"
Nov 03 01:33:50.115153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.115171    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b"} err="failed to get container status \"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b\": rpc error: code = NotFound desc = could not find container \"c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b\": container with ID starting with c8f02a08f4e4c2d2a2825ffd934a03f5746564804f31caeb872db29b61207b0b not found: ID does not exist"
Nov 03 01:33:50.132230 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.127385    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:33:50.229185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.228877    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5bfc51-86f8495ff9-wgnll"]
Nov 03 01:33:50.906200 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:33:50.906150    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" path="/var/lib/kubelet/pods/adc1cf63-4645-4a69-ae98-5fa2dafa7fbd/volumes"
Nov 03 01:34:09.879930 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:09.879848    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:34:28.218720 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.218679    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:34:28.219713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219689    4531 cpu_manager.go:414] "containerMap: removing container" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219714    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408dcbcb-fb13-4ccf-a936-8d486fcc11f5" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219735    4531 cpu_manager.go:414] "containerMap: removing container" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219742    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b335e2-51de-4e95-85f2-75b256c50838" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219776    4531 cpu_manager.go:414] "containerMap: removing container" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219785    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219807    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy"
Nov 03 01:34:28.219829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219813    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219833    4531 cpu_manager.go:414] "containerMap: removing container" podUID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" containerName="payload-processing"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219840    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" containerName="payload-processing"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219862    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219870    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219900    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219908    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e12344-17f1-4dc7-bd43-176d0f99b2c9" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219925    4531 cpu_manager.go:414] "containerMap: removing container" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.219934    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220111    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="429ff11e-60e8-4d41-a6e3-c2981e342b1b" containerName="istio-proxy"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220129    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b22bb9-80bf-460b-836e-2e8af3b30d83" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220138    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f53f02-45ad-44bd-bf1e-f519f5332df2" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220152    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdc6cdc-b2f6-4e3e-b02e-a985ce357fa8" containerName="cleanup"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220237    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6018817e-d13d-42cd-b420-3a15c387c78a" containerName="istio-proxy"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220251    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc1cf63-4645-4a69-ae98-5fa2dafa7fbd" containerName="payload-processing"
Nov 03 01:34:28.220258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.220266    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f6310c-f0ea-44c2-8109-7ba8d241b3f1" containerName="cleanup"
Nov 03 01:34:28.224415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.224390    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.278113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.278067    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:34:28.325467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.325439    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-737bf2d8-openshift-default-dockercfg-7f9q9\""
Nov 03 01:34:28.369423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369381    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369681 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369498    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369681 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369645    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369726    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369893 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q8x\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.369965 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.370024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.369988    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.370081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.370063    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472028 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.471927    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472028 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472017    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q8x\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472070    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472233    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472310    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472407    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472421    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472439    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472439    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472794 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472605    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472794 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472744    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.472960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.472937    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.473092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.473067    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.475123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.475100    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.475330 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.475307    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.481500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.481465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.482120 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.482098    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q8x\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.578110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.578072    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:28.890914 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:34:28.890882    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1660361f_e8b7_43df_bd7b_3950dfa5e88d.slice/crio-bf05870e8cd6a3142c74b71a06e5ba000013af3afc1717963b8e1593c8524253 WatchSource:0}: Error finding container bf05870e8cd6a3142c74b71a06e5ba000013af3afc1717963b8e1593c8524253: Status 404 returned error can't find the container with id bf05870e8cd6a3142c74b71a06e5ba000013af3afc1717963b8e1593c8524253
Nov 03 01:34:28.891067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.890963    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:34:28.894336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.894300    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:34:28.894429 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.894380    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:34:28.894429 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:28.894414    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:34:29.535180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:29.535125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" event={"ID":"1660361f-e8b7-43df-bd7b-3950dfa5e88d","Type":"ContainerStarted","Data":"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"}
Nov 03 01:34:29.535685 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:29.535272    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" event={"ID":"1660361f-e8b7-43df-bd7b-3950dfa5e88d","Type":"ContainerStarted","Data":"bf05870e8cd6a3142c74b71a06e5ba000013af3afc1717963b8e1593c8524253"}
Nov 03 01:34:29.559094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:29.559032    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podStartSLOduration=1.559016349 podStartE2EDuration="1.559016349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:29.557318012 +0000 UTC m=+3214.771061687" watchObservedRunningTime="2026-11-03 01:34:29.559016349 +0000 UTC m=+3214.772760021"
Nov 03 01:34:29.578592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:29.578557    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:30.158348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.158302    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:34:30.158570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.158519    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" containerID="cri-o://7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f" gracePeriod=30
Nov 03 01:34:30.581018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.580988    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 01:34:30.581522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.581043    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 01:34:30.701546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.701491    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:34:30.701761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.701562    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:34:30.972664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.972577    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:34:30.972839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:30.972777    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy" containerID="cri-o://8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077" gracePeriod=30
Nov 03 01:34:31.578889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:31.578853    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 01:34:31.579095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:31.578926    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 01:34:32.605195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:32.590274    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 01:34:32.605195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:32.590339    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 01:34:33.579128 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:33.579091    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 01:34:33.579372 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:33.579154    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 01:34:34.156197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:34.152918    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 01:34:34.156197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:34.153522    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" containerID="cri-o://aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79" gracePeriod=30
Nov 03 01:34:34.579209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:34.579148    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 03 01:34:34.579397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:34.579236    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 03 01:34:35.474746 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.474718    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:34:35.550345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550302    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550552 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550406    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550552 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550495    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550667    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:35.550782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550710    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550745    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550894 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550829    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550894 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550872    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550936    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.550999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.550969    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\") pod \"a667d4b4-e891-4402-98d5-55a6edb85f51\" (UID: \"a667d4b4-e891-4402-98d5-55a6edb85f51\") "
Nov 03 01:34:35.552708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.551855    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.552708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.552484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:35.552708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.552566    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:35.552963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.552870    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:35.552963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.552877    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:35.555704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.555679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "kube-api-access-sq7mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:35.556516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.556463    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:35.557039 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.557003    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:35.557136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.557070    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo" pod "a667d4b4-e891-4402-98d5-55a6edb85f51" (UID: "a667d4b4-e891-4402-98d5-55a6edb85f51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:34:35.583680 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.583648    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:35.584207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.584144    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:35.585234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.585212    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655039    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655076    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655091    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655104    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655117    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7mx\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-kube-api-access-sq7mx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655131    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655146    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a667d4b4-e891-4402-98d5-55a6edb85f51-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.655782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.655224    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a667d4b4-e891-4402-98d5-55a6edb85f51-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:35.760875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.760791    4531 generic.go:356] "Generic (PLEG): container finished" podID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerID="7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f" exitCode=0
Nov 03 01:34:35.761293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.761134    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"
Nov 03 01:34:35.761654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.761136    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" event={"ID":"a667d4b4-e891-4402-98d5-55a6edb85f51","Type":"ContainerDied","Data":"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"}
Nov 03 01:34:35.761868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.761849    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c" event={"ID":"a667d4b4-e891-4402-98d5-55a6edb85f51","Type":"ContainerDied","Data":"efb36d14926ab94ee5f3bf08a806326b35024a486b6e9a8dc733ea8806c0519c"}
Nov 03 01:34:35.762023 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.762007    4531 scope.go:122] "RemoveContainer" containerID="7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"
Nov 03 01:34:35.779345 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.779320    4531 scope.go:122] "RemoveContainer" containerID="7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"
Nov 03 01:34:35.779937 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:35.779912    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f\": container with ID starting with 7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f not found: ID does not exist" containerID="7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"
Nov 03 01:34:35.780094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.780070    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f"} err="failed to get container status \"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f\": rpc error: code = NotFound desc = could not find container \"7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f\": container with ID starting with 7e2ba759dc3c088e211d5888fb87d71450e8bde5cacf2754317ab921ad19e24f not found: ID does not exist"
Nov 03 01:34:35.789869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.789846    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:34:35.796224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:35.796152    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-af8244-openshift-default-594d5d565c-l4j5c"]
Nov 03 01:34:36.318334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.318307    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:34:36.366673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366583    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.366673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366657    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.366914 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366817    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.366974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366923    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.366974 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366940    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:36.367087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.366976    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.367087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367023    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.367263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367128    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.367263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367191    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.367263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367222    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\") pod \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\" (UID: \"3aba7506-9c5f-4bef-baf1-66b4a98065bc\") "
Nov 03 01:34:36.367500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367473    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:36.367748 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367720    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:36.367841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.367783    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:36.368079 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.368051    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:36.368252 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.368221    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.368408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.368349    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.368408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.368373    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.368408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.368390    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.370307 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.370275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:36.370677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.370650    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "kube-api-access-8hmp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:36.370773 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.370680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:36.372453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.372427    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo" pod "3aba7506-9c5f-4bef-baf1-66b4a98065bc" (UID: "3aba7506-9c5f-4bef-baf1-66b4a98065bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:34:36.478184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.474505    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.478184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.474547    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hmp8\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-kube-api-access-8hmp8\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.478184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.474563    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.478184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.474576    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aba7506-9c5f-4bef-baf1-66b4a98065bc-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.478184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.474590    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aba7506-9c5f-4bef-baf1-66b4a98065bc-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:36.779860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.779751    4531 generic.go:356] "Generic (PLEG): container finished" podID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerID="8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077" exitCode=0
Nov 03 01:34:36.779860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.779807    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"
Nov 03 01:34:36.779860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.779838    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" event={"ID":"3aba7506-9c5f-4bef-baf1-66b4a98065bc","Type":"ContainerDied","Data":"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"}
Nov 03 01:34:36.780203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.779901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4" event={"ID":"3aba7506-9c5f-4bef-baf1-66b4a98065bc","Type":"ContainerDied","Data":"5d60d838212e79ad630568f4484abbf1ec6dbeb9a1607607b9c5870114447a10"}
Nov 03 01:34:36.780203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.779932    4531 scope.go:122] "RemoveContainer" containerID="8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"
Nov 03 01:34:36.795807 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.795780    4531 scope.go:122] "RemoveContainer" containerID="8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"
Nov 03 01:34:36.796265 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:36.796234    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077\": container with ID starting with 8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077 not found: ID does not exist" containerID="8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"
Nov 03 01:34:36.796349 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.796278    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077"} err="failed to get container status \"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077\": rpc error: code = NotFound desc = could not find container \"8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077\": container with ID starting with 8f20b29944eb3518d2b64c95301c2b8576de52ba6e6bd97594e8b36db1cfe077 not found: ID does not exist"
Nov 03 01:34:36.809319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.809291    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:34:36.813522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.813497    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3c5983-openshift-default-86c5bc855c-jtjq4"]
Nov 03 01:34:36.904882 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.904848    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" path="/var/lib/kubelet/pods/3aba7506-9c5f-4bef-baf1-66b4a98065bc/volumes"
Nov 03 01:34:36.905409 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:36.905395    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" path="/var/lib/kubelet/pods/a667d4b4-e891-4402-98d5-55a6edb85f51/volumes"
Nov 03 01:34:38.768211 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:38.768149    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:34:38.768666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:38.768249    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:34:39.488293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.488263    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 01:34:39.627268 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627229    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627306    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627390    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627433    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627483    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627549    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627639    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627706    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.627786 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627543    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:39.627888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627806    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\") pod \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\" (UID: \"9bb4d203-a9f5-4ffe-8e40-fd903d42f564\") "
Nov 03 01:34:39.628027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.627999    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:39.628237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628215    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:39.628325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628249    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:39.628381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628339    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:39.628709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628682    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.628812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628714    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.628812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628732    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.628812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628748    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.628812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.628761    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.629989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.629963    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:39.630560 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.630538    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:39.630656 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.630591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "kube-api-access-j857d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:39.630829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.630811    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo" pod "9bb4d203-a9f5-4ffe-8e40-fd903d42f564" (UID: "9bb4d203-a9f5-4ffe-8e40-fd903d42f564"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:34:39.732971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.732870    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.732971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.732914    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.732971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.732927    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.732971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.732940    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j857d\" (UniqueName: \"kubernetes.io/projected/9bb4d203-a9f5-4ffe-8e40-fd903d42f564-kube-api-access-j857d\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:39.808787 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.808736    4531 generic.go:356] "Generic (PLEG): container finished" podID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerID="aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79" exitCode=0
Nov 03 01:34:39.809286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.808971    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" event={"ID":"9bb4d203-a9f5-4ffe-8e40-fd903d42f564","Type":"ContainerDied","Data":"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"}
Nov 03 01:34:39.809286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.809069    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd" event={"ID":"9bb4d203-a9f5-4ffe-8e40-fd903d42f564","Type":"ContainerDied","Data":"51dc905378e8ebfd6f0a125b19a5312a585a6d3533609d0b9d352529438cbb35"}
Nov 03 01:34:39.809286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.809107    4531 scope.go:122] "RemoveContainer" containerID="aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"
Nov 03 01:34:39.809450 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.809321    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"
Nov 03 01:34:39.831976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.831952    4531 scope.go:122] "RemoveContainer" containerID="aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"
Nov 03 01:34:39.832358 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:39.832336    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79\": container with ID starting with aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79 not found: ID does not exist" containerID="aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"
Nov 03 01:34:39.832435 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.832366    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79"} err="failed to get container status \"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79\": rpc error: code = NotFound desc = could not find container \"aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79\": container with ID starting with aa25646a0ca65053d8aef79d1a889cd6633f6cb560a7a6e4245280dbc040cb79 not found: ID does not exist"
Nov 03 01:34:39.845888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.845854    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 01:34:39.850578 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:39.850546    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-ss9cd"]
Nov 03 01:34:39.879316 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:39.879280    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:34:40.905895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:40.905861    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" path="/var/lib/kubelet/pods/9bb4d203-a9f5-4ffe-8e40-fd903d42f564/volumes"
Nov 03 01:34:43.789594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:43.789562    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:34:43.790011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:43.789758    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" containerID="cri-o://6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03" gracePeriod=30
Nov 03 01:34:46.597674 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:46.597624    4531 patch_prober.go:29] interesting pod/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:34:46.598185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:46.597692    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:34:49.093191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.093147    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:34:49.252957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.252917    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.252985    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253052    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253086    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253112    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253184    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253247    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253295    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:49.253620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\") pod \"37be08c0-1f71-43a8-869e-bc918ba4088c\" (UID: \"37be08c0-1f71-43a8-869e-bc918ba4088c\") "
Nov 03 01:34:49.253673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253588    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:49.254058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.253967    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:49.254245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:49.254325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254246    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:49.254523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254504    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.254563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254530    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.254563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254545    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.254563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254559    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37be08c0-1f71-43a8-869e-bc918ba4088c-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.254657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.254574    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.255861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.255833    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:49.255861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.255848    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:49.256352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.256330    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:34:49.256714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.256697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8" pod "37be08c0-1f71-43a8-869e-bc918ba4088c" (UID: "37be08c0-1f71-43a8-869e-bc918ba4088c"). InnerVolumeSpecName "kube-api-access-7qrn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:49.356469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.356433    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrn8\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-kube-api-access-7qrn8\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.356469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.356464    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.356469 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.356475    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.356694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.356483    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37be08c0-1f71-43a8-869e-bc918ba4088c-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:49.900281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.899459    4531 generic.go:356] "Generic (PLEG): container finished" podID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerID="6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03" exitCode=0
Nov 03 01:34:49.900281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.899650    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" event={"ID":"37be08c0-1f71-43a8-869e-bc918ba4088c","Type":"ContainerDied","Data":"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"}
Nov 03 01:34:49.900281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.899681    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc" event={"ID":"37be08c0-1f71-43a8-869e-bc918ba4088c","Type":"ContainerDied","Data":"dcd828d33d491d7c5fa06e60363d4abb3ffd074e816bc1b43825256668f197f7"}
Nov 03 01:34:49.900281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.899699    4531 scope.go:122] "RemoveContainer" containerID="6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"
Nov 03 01:34:49.900281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.899808    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"
Nov 03 01:34:49.919406 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.919341    4531 scope.go:122] "RemoveContainer" containerID="6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"
Nov 03 01:34:49.919824 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:34:49.919795    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03\": container with ID starting with 6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03 not found: ID does not exist" containerID="6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"
Nov 03 01:34:49.919944 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.919838    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03"} err="failed to get container status \"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03\": rpc error: code = NotFound desc = could not find container \"6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03\": container with ID starting with 6b7d90283d64d32815f0604c02c0b9bee6dbfe7269146024420dfce49c774e03 not found: ID does not exist"
Nov 03 01:34:49.957229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.957182    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:34:49.963509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:49.963472    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1f08d-openshift-default-6b4b5fc8b9-tljzc"]
Nov 03 01:34:51.111186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:51.096282    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" path="/var/lib/kubelet/pods/37be08c0-1f71-43a8-869e-bc918ba4088c/volumes"
Nov 03 01:34:52.544191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:34:52.529989    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:35:04.809924 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:04.809883    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:35:09.810553 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:09.810479    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" podUID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" containerName="payload-processing" containerID="cri-o://34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb" gracePeriod=25
Nov 03 01:35:09.883413 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:35:09.883374    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:35:10.124603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.124577    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:35:10.234222 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.234185    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\") pod \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") "
Nov 03 01:35:10.234431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.234259    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\" (UniqueName: \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\") pod \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\" (UID: \"8b24bea7-3fe6-423f-88cf-7673f98e0b54\") "
Nov 03 01:35:10.234751 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.234673    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume" pod "8b24bea7-3fe6-423f-88cf-7673f98e0b54" (UID: "8b24bea7-3fe6-423f-88cf-7673f98e0b54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:10.235610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.235577    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24bea7-3fe6-423f-88cf-7673f98e0b54-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:10.236960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.236925    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6" pod "8b24bea7-3fe6-423f-88cf-7673f98e0b54" (UID: "8b24bea7-3fe6-423f-88cf-7673f98e0b54"). InnerVolumeSpecName "kube-api-access-c8xg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:10.337570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.337523    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xg6\" (UniqueName: \"kubernetes.io/projected/8b24bea7-3fe6-423f-88cf-7673f98e0b54-kube-api-access-c8xg6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:10.531543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.531458    4531 generic.go:356] "Generic (PLEG): container finished" podID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" containerID="34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb" exitCode=0
Nov 03 01:35:10.531543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.531519    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"
Nov 03 01:35:10.531761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.531543    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" event={"ID":"8b24bea7-3fe6-423f-88cf-7673f98e0b54","Type":"ContainerDied","Data":"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"}
Nov 03 01:35:10.531761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.531609    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr" event={"ID":"8b24bea7-3fe6-423f-88cf-7673f98e0b54","Type":"ContainerDied","Data":"33a4adc6763f3e8db90b1b0c7d45ff6ccda5eabe56127a9b087794826a55edaa"}
Nov 03 01:35:10.531761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.531642    4531 scope.go:122] "RemoveContainer" containerID="34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"
Nov 03 01:35:10.547366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.547342    4531 scope.go:122] "RemoveContainer" containerID="34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"
Nov 03 01:35:10.547983 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:35:10.547958    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb\": container with ID starting with 34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb not found: ID does not exist" containerID="34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"
Nov 03 01:35:10.548067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.547995    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb"} err="failed to get container status \"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb\": rpc error: code = NotFound desc = could not find container \"34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb\": container with ID starting with 34cae99f766c1d60a598c555a14d22099298b9c47d4e7284cbf75b223e34e2eb not found: ID does not exist"
Nov 03 01:35:10.563565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.563534    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:35:10.588203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.579370    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6afd6-786d97f6c5-cndzr"]
Nov 03 01:35:10.948191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:10.939969    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" path="/var/lib/kubelet/pods/8b24bea7-3fe6-423f-88cf-7673f98e0b54/volumes"
Nov 03 01:35:13.293185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.291717    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305498    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305529    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305592    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305600    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305616    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305623    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305641    4531 cpu_manager.go:414] "containerMap: removing container" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305650    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305702    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" containerName="payload-processing"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305711    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" containerName="payload-processing"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305934    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b24bea7-3fe6-423f-88cf-7673f98e0b54" containerName="payload-processing"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305955    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a667d4b4-e891-4402-98d5-55a6edb85f51" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305978    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aba7506-9c5f-4bef-baf1-66b4a98065bc" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.305989    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb4d203-a9f5-4ffe-8e40-fd903d42f564" containerName="istio-proxy"
Nov 03 01:35:13.306185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.306002    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="37be08c0-1f71-43a8-869e-bc918ba4088c" containerName="istio-proxy"
Nov 03 01:35:13.339233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.339188    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:35:13.339441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.339336    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.398983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399137    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399177    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399203    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv57l\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399420    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.399608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.399448    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501124    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501228    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501265    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501297    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv57l\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501394    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501659    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501714 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501726    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501828 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501810    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.501998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.501828    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.502357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.502323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.504090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.504066    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.504251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.504203    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.510216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.510194    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.510678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.510654    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv57l\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.683886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.683853    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:13.989941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.989910    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:35:13.991466 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:35:13.991437    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6cea55_70d2_4197_8d23_72c494a1e4a1.slice/crio-86c739022889654563a094ee2e0b76ecb3081272da7aa50c1d8b6ed339989941 WatchSource:0}: Error finding container 86c739022889654563a094ee2e0b76ecb3081272da7aa50c1d8b6ed339989941: Status 404 returned error can't find the container with id 86c739022889654563a094ee2e0b76ecb3081272da7aa50c1d8b6ed339989941
Nov 03 01:35:13.995055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.995015    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:13.995145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.995106    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:13.995196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:13.995148    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:14.612188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:14.606415    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" event={"ID":"1f6cea55-70d2-4197-8d23-72c494a1e4a1","Type":"ContainerStarted","Data":"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"}
Nov 03 01:35:14.612188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:14.606490    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" event={"ID":"1f6cea55-70d2-4197-8d23-72c494a1e4a1","Type":"ContainerStarted","Data":"86c739022889654563a094ee2e0b76ecb3081272da7aa50c1d8b6ed339989941"}
Nov 03 01:35:14.686350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:14.686253    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:15.686465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:15.686429    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 03 01:35:15.687040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:15.686486    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 03 01:35:16.684742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:16.684699    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 03 01:35:16.684943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:16.684763    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 03 01:35:17.684926 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:17.684889    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 03 01:35:17.685364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:17.684946    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 03 01:35:18.684871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:18.684831    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 03 01:35:18.685086 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:18.684908    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 03 01:35:19.685171 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:19.685133    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Nov 03 01:35:19.685675 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:19.685218    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Nov 03 01:35:20.690432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:20.690400    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:20.690857 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:20.690755    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:20.691382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:20.691362    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:35:20.716769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:20.716722    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podStartSLOduration=7.716709114 podStartE2EDuration="7.716709114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:14.657430076 +0000 UTC m=+3259.871173750" watchObservedRunningTime="2026-11-03 01:35:20.716709114 +0000 UTC m=+3265.930452785"
Nov 03 01:35:26.920537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:26.920504    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:35:28.338507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.338462    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:35:28.343093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.343066    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.369260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.369230    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:35:28.400221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.400107    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2cb\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.400667 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.400625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.400815 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.400800    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.400960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.400946    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.401090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.401076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.401394 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.401372    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.401638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.401601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.401767 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.401740    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.401832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.401787    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503301    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503399    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503433    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503542 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503514    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2cb\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503725 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503725 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503587    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503725 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503725 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503694    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503907 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.503956 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.503923    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.504398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.504371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.504903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.504427    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.504903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.504527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.504903 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.504632    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.506391 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.506363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.506499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.506365    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.512123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.512094    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.512303 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.512283    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2cb\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\") pod \"e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") " pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.658827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.658717    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:28.949303 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:35:28.949265    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf784a2e6_e82e_4c45_a36c_6c1d3f10b640.slice/crio-bdc6547f9382722db2451aa7137dadfc4acae450f6a9ecd51fe6ed0660d1be51 WatchSource:0}: Error finding container bdc6547f9382722db2451aa7137dadfc4acae450f6a9ecd51fe6ed0660d1be51: Status 404 returned error can't find the container with id bdc6547f9382722db2451aa7137dadfc4acae450f6a9ecd51fe6ed0660d1be51
Nov 03 01:35:28.950923 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.950888    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:35:28.952602 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.952570    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:28.952729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.952638    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:28.952729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:28.952667    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:35:29.754198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:29.754133    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:35:29.754734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:29.754417    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" podUID="3a53354e-580b-4411-98d1-46e59aa3ab56" containerName="maas-api" containerID="cri-o://ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc" gracePeriod=30
Nov 03 01:35:30.014430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.010713    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" event={"ID":"f784a2e6-e82e-4c45-a36c-6c1d3f10b640","Type":"ContainerStarted","Data":"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"}
Nov 03 01:35:30.014430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.010782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" event={"ID":"f784a2e6-e82e-4c45-a36c-6c1d3f10b640","Type":"ContainerStarted","Data":"bdc6547f9382722db2451aa7137dadfc4acae450f6a9ecd51fe6ed0660d1be51"}
Nov 03 01:35:30.477917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.477890    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:35:30.496490 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.496430    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podStartSLOduration=2.496412311 podStartE2EDuration="2.496412311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:30.060555362 +0000 UTC m=+3275.274299035" watchObservedRunningTime="2026-11-03 01:35:30.496412311 +0000 UTC m=+3275.710155984"
Nov 03 01:35:30.525577 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.525538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\" (UniqueName: \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\") pod \"3a53354e-580b-4411-98d1-46e59aa3ab56\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") "
Nov 03 01:35:30.525770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.525664    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\") pod \"3a53354e-580b-4411-98d1-46e59aa3ab56\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") "
Nov 03 01:35:30.525770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.525688    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\") pod \"3a53354e-580b-4411-98d1-46e59aa3ab56\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") "
Nov 03 01:35:30.525770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.525755    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\") pod \"3a53354e-580b-4411-98d1-46e59aa3ab56\" (UID: \"3a53354e-580b-4411-98d1-46e59aa3ab56\") "
Nov 03 01:35:30.526048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.526021    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca" pod "3a53354e-580b-4411-98d1-46e59aa3ab56" (UID: "3a53354e-580b-4411-98d1-46e59aa3ab56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:30.530286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.526385    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a53354e-580b-4411-98d1-46e59aa3ab56-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:30.569720 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.569649    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs" pod "3a53354e-580b-4411-98d1-46e59aa3ab56" (UID: "3a53354e-580b-4411-98d1-46e59aa3ab56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:35:30.570424 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.570385    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94" pod "3a53354e-580b-4411-98d1-46e59aa3ab56" (UID: "3a53354e-580b-4411-98d1-46e59aa3ab56"). InnerVolumeSpecName "kube-api-access-whv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:30.571686 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.571655    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls" pod "3a53354e-580b-4411-98d1-46e59aa3ab56" (UID: "3a53354e-580b-4411-98d1-46e59aa3ab56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:35:30.635494 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.635448    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:30.635761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.635741    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a53354e-580b-4411-98d1-46e59aa3ab56-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:30.635873 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.635859    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whv94\" (UniqueName: \"kubernetes.io/projected/3a53354e-580b-4411-98d1-46e59aa3ab56-kube-api-access-whv94\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:30.660189 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.659736    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:30.661806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.661718    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 01:35:30.661806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:30.661787    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 01:35:31.040901 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.040862    4531 generic.go:356] "Generic (PLEG): container finished" podID="3a53354e-580b-4411-98d1-46e59aa3ab56" containerID="ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc" exitCode=0
Nov 03 01:35:31.042782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.042757    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"
Nov 03 01:35:31.043792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.042751    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" event={"ID":"3a53354e-580b-4411-98d1-46e59aa3ab56","Type":"ContainerDied","Data":"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"}
Nov 03 01:35:31.043945 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.043923    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58" event={"ID":"3a53354e-580b-4411-98d1-46e59aa3ab56","Type":"ContainerDied","Data":"51900d5d6315e96ea8d4e1b207e0d28cf7f7d1521653df22da12cf70b8f5d650"}
Nov 03 01:35:31.044027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.043993    4531 scope.go:122] "RemoveContainer" containerID="ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"
Nov 03 01:35:31.066442 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.066412    4531 scope.go:122] "RemoveContainer" containerID="ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"
Nov 03 01:35:31.067030 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:35:31.067003    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc\": container with ID starting with ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc not found: ID does not exist" containerID="ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"
Nov 03 01:35:31.067309 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.067278    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc"} err="failed to get container status \"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc\": rpc error: code = NotFound desc = could not find container \"ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc\": container with ID starting with ddc4862d786012004dbc59a3f69ea9a01235e055d85421318a1b90e4e370c9cc not found: ID does not exist"
Nov 03 01:35:31.075751 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.075724    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:35:31.080051 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.080025    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b4971a-5b8d6ff4d7-2zv58"]
Nov 03 01:35:31.659794 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.659753    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 01:35:31.659976 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.659808    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 01:35:31.921500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:31.921421    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" podUID="eb053be2-0283-41dc-aab5-8bc980205edb" containerName="payload-processing" containerID="cri-o://da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b" gracePeriod=25
Nov 03 01:35:32.326225 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.326139    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:35:32.420363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.418632    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\" (UniqueName: \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\") pod \"eb053be2-0283-41dc-aab5-8bc980205edb\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") "
Nov 03 01:35:32.420363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.418709    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\") pod \"eb053be2-0283-41dc-aab5-8bc980205edb\" (UID: \"eb053be2-0283-41dc-aab5-8bc980205edb\") "
Nov 03 01:35:32.420363 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.419982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume" pod "eb053be2-0283-41dc-aab5-8bc980205edb" (UID: "eb053be2-0283-41dc-aab5-8bc980205edb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:32.422316 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.422283    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd" pod "eb053be2-0283-41dc-aab5-8bc980205edb" (UID: "eb053be2-0283-41dc-aab5-8bc980205edb"). InnerVolumeSpecName "kube-api-access-6tbbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:32.521018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.520943    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tbbd\" (UniqueName: \"kubernetes.io/projected/eb053be2-0283-41dc-aab5-8bc980205edb-kube-api-access-6tbbd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:32.521018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.520972    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb053be2-0283-41dc-aab5-8bc980205edb-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:32.659707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.659665    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 01:35:32.659877 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.659722    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 01:35:32.905507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:32.905473    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a53354e-580b-4411-98d1-46e59aa3ab56" path="/var/lib/kubelet/pods/3a53354e-580b-4411-98d1-46e59aa3ab56/volumes"
Nov 03 01:35:33.073897 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.073863    4531 generic.go:356] "Generic (PLEG): container finished" podID="eb053be2-0283-41dc-aab5-8bc980205edb" containerID="da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b" exitCode=0
Nov 03 01:35:33.074098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.073923    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"
Nov 03 01:35:33.074098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.073919    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" event={"ID":"eb053be2-0283-41dc-aab5-8bc980205edb","Type":"ContainerDied","Data":"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"}
Nov 03 01:35:33.074098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.074061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t" event={"ID":"eb053be2-0283-41dc-aab5-8bc980205edb","Type":"ContainerDied","Data":"80245403c0b3cf43fbf80908e2ed10c2b9375d859d9d7c81be82aaaaf8524b1b"}
Nov 03 01:35:33.074098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.074095    4531 scope.go:122] "RemoveContainer" containerID="da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"
Nov 03 01:35:33.086344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.086321    4531 scope.go:122] "RemoveContainer" containerID="da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"
Nov 03 01:35:33.086651 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:35:33.086630    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b\": container with ID starting with da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b not found: ID does not exist" containerID="da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"
Nov 03 01:35:33.086713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.086663    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b"} err="failed to get container status \"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b\": rpc error: code = NotFound desc = could not find container \"da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b\": container with ID starting with da73a8824acc15b39a50e826e33136309566e1044c87ce6cb06040b6dd52819b not found: ID does not exist"
Nov 03 01:35:33.110319 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.110282    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:35:33.114546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.114520    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4971a-558c5458f5-jn77t"]
Nov 03 01:35:33.664637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.664598    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 01:35:33.665199 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:33.665150    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 01:35:34.659568 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:34.659526    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 01:35:34.659812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:34.659590    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 01:35:34.905669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:34.905635    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb053be2-0283-41dc-aab5-8bc980205edb" path="/var/lib/kubelet/pods/eb053be2-0283-41dc-aab5-8bc980205edb/volumes"
Nov 03 01:35:35.839003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:35.838973    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:35.839991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:35.839949    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:35.842747 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:35.842723    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:35:39.880001 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:35:39.879899    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:35:44.215511 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:44.215477    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:35:44.259065 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:44.259026    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:35:49.216320 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:49.216259    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing" containerID="cri-o://54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b" gracePeriod=25
Nov 03 01:35:49.259841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:49.259802    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing" containerID="cri-o://6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db" gracePeriod=25
Nov 03 01:35:56.044607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.044557    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:35:56.045127 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.044844    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" podUID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" containerName="maas-api" containerID="cri-o://2416ac6595d493530865e457cc100c3328cecda896051645bda35d4fee474708" gracePeriod=30
Nov 03 01:35:56.490293 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.476375    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:56.522256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.521951    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:56.572589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.572538    4531 generic.go:356] "Generic (PLEG): container finished" podID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" containerID="2416ac6595d493530865e457cc100c3328cecda896051645bda35d4fee474708" exitCode=0
Nov 03 01:35:56.572781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.572660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" event={"ID":"18ff3ca3-1d34-4469-ae60-1486f13d2f94","Type":"ContainerDied","Data":"2416ac6595d493530865e457cc100c3328cecda896051645bda35d4fee474708"}
Nov 03 01:35:56.874420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.874391    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:35:56.995603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.994925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\") pod \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") "
Nov 03 01:35:56.995603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.995088    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\") pod \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") "
Nov 03 01:35:56.995863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.995778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\" (UniqueName: \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\") pod \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") "
Nov 03 01:35:56.995863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.995807    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\") pod \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\" (UID: \"18ff3ca3-1d34-4469-ae60-1486f13d2f94\") "
Nov 03 01:35:56.996383 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.996357    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca" pod "18ff3ca3-1d34-4469-ae60-1486f13d2f94" (UID: "18ff3ca3-1d34-4469-ae60-1486f13d2f94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:56.998045 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.998014    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls" pod "18ff3ca3-1d34-4469-ae60-1486f13d2f94" (UID: "18ff3ca3-1d34-4469-ae60-1486f13d2f94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:35:56.998145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.998108    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j" pod "18ff3ca3-1d34-4469-ae60-1486f13d2f94" (UID: "18ff3ca3-1d34-4469-ae60-1486f13d2f94"). InnerVolumeSpecName "kube-api-access-v6r8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:56.998353 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:56.998335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs" pod "18ff3ca3-1d34-4469-ae60-1486f13d2f94" (UID: "18ff3ca3-1d34-4469-ae60-1486f13d2f94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:35:57.097925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.097843    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6r8j\" (UniqueName: \"kubernetes.io/projected/18ff3ca3-1d34-4469-ae60-1486f13d2f94-kube-api-access-v6r8j\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:57.097925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.097883    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:57.097925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.097899    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ff3ca3-1d34-4469-ae60-1486f13d2f94-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:57.097925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.097911    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18ff3ca3-1d34-4469-ae60-1486f13d2f94-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:57.584406 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.584353    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486" event={"ID":"18ff3ca3-1d34-4469-ae60-1486f13d2f94","Type":"ContainerDied","Data":"aa0c9307e238bf24b75584edd3b706db2991b600d33ae6a655070a76c755ac5b"}
Nov 03 01:35:57.584616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.584434    4531 scope.go:122] "RemoveContainer" containerID="2416ac6595d493530865e457cc100c3328cecda896051645bda35d4fee474708"
Nov 03 01:35:57.584616 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.584580    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"
Nov 03 01:35:57.726193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.712797    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:35:57.726193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:57.716254    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-737bf2d8-7cb8cc6f49-pc486"]
Nov 03 01:35:58.906556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:35:58.906517    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" path="/var/lib/kubelet/pods/18ff3ca3-1d34-4469-ae60-1486f13d2f94/volumes"
Nov 03 01:36:01.917255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.917219    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:36:01.917709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.917297    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:36:01.943203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.938096    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" containerID="cri-o://636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c" gracePeriod=30
Nov 03 01:36:01.943203 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.938291    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" containerID="cri-o://bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb" gracePeriod=30
Nov 03 01:36:01.952315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.952286    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:36:01.952766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:01.952713    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" containerID="cri-o://20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6" gracePeriod=30
Nov 03 01:36:05.586080 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.586045    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:36:05.586527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.586119    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:36:05.692807 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.692771    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:36:05.692971 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.692831    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:36:05.842527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.842431    4531 patch_prober.go:29] interesting pod/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:36:05.842692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:05.842553    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:36:06.370550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:06.370495    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:36:06.373227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:06.373193    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:36:07.285362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.285329    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:36:07.394570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.394543    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:36:07.400282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.400255    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:36:07.421268 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421233    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.421446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421364    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.421446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421414    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421445    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421546    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421578    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421607    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.421737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421734    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.422204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.421768    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.422204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422038    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.422204 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422096    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:07.422380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.422499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422447    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.422589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422518    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.422776 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.422752    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.423023 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423002    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.423312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423289    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.423470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\") pod \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\" (UID: \"1f6cea55-70d2-4197-8d23-72c494a1e4a1\") "
Nov 03 01:36:07.423470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423372    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.423470 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423398    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.423744 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423713    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:07.423812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423776    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.423928 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423906    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.423992 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423915    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.423992 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.423972    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.424110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424027    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.424110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424081    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.424352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424177    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.424352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.424352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424274    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\") pod \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\" (UID: \"1660361f-e8b7-43df-bd7b-3950dfa5e88d\") "
Nov 03 01:36:07.424352 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.424339    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\") pod \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\" (UID: \"f784a2e6-e82e-4c45-a36c-6c1d3f10b640\") "
Nov 03 01:36:07.425129 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425102    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425134    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425139    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425149    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425201    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425220    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425234    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425256 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425248    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425262    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425276    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.425648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425347    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.425648 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.425521    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.426340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.426312    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.426430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.426350    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:07.428865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.428839    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.432014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.430871    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:36:07.432014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.430985    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "kube-api-access-gv57l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.432014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.431199    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434028    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434033    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434105    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token" pod "1f6cea55-70d2-4197-8d23-72c494a1e4a1" (UID: "1f6cea55-70d2-4197-8d23-72c494a1e4a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434130    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.434244 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434138    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:36:07.434631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.434290    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb" pod "f784a2e6-e82e-4c45-a36c-6c1d3f10b640" (UID: "f784a2e6-e82e-4c45-a36c-6c1d3f10b640"). InnerVolumeSpecName "kube-api-access-2h2cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.436100 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.436048    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "kube-api-access-s6q8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:07.436100 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.436048    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy" pod "1660361f-e8b7-43df-bd7b-3950dfa5e88d" (UID: "1660361f-e8b7-43df-bd7b-3950dfa5e88d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531632    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531679    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531694    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531709    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531722    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531735    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531748    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531761    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531774    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv57l\" (UniqueName: \"kubernetes.io/projected/1f6cea55-70d2-4197-8d23-72c494a1e4a1-kube-api-access-gv57l\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531788    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531800    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.532347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531813    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.531826    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2cb\" (UniqueName: \"kubernetes.io/projected/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-kube-api-access-2h2cb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.532523    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1660361f-e8b7-43df-bd7b-3950dfa5e88d-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.532540    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.532555    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.532568    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f784a2e6-e82e-4c45-a36c-6c1d3f10b640-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.533062 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.532582    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q8x\" (UniqueName: \"kubernetes.io/projected/1660361f-e8b7-43df-bd7b-3950dfa5e88d-kube-api-access-s6q8x\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:07.714582 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.714548    4531 generic.go:356] "Generic (PLEG): container finished" podID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerID="20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6" exitCode=0
Nov 03 01:36:07.714775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.714598    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"
Nov 03 01:36:07.714775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.714633    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" event={"ID":"f784a2e6-e82e-4c45-a36c-6c1d3f10b640","Type":"ContainerDied","Data":"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"}
Nov 03 01:36:07.714775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.714721    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm" event={"ID":"f784a2e6-e82e-4c45-a36c-6c1d3f10b640","Type":"ContainerDied","Data":"bdc6547f9382722db2451aa7137dadfc4acae450f6a9ecd51fe6ed0660d1be51"}
Nov 03 01:36:07.714775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.714752    4531 scope.go:122] "RemoveContainer" containerID="20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"
Nov 03 01:36:07.716425 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.716403    4531 generic.go:356] "Generic (PLEG): container finished" podID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerID="bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb" exitCode=0
Nov 03 01:36:07.716559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.716449    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"
Nov 03 01:36:07.716559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.716478    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" event={"ID":"1660361f-e8b7-43df-bd7b-3950dfa5e88d","Type":"ContainerDied","Data":"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"}
Nov 03 01:36:07.716559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.716523    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89" event={"ID":"1660361f-e8b7-43df-bd7b-3950dfa5e88d","Type":"ContainerDied","Data":"bf05870e8cd6a3142c74b71a06e5ba000013af3afc1717963b8e1593c8524253"}
Nov 03 01:36:07.718236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.718152    4531 generic.go:356] "Generic (PLEG): container finished" podID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerID="636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c" exitCode=0
Nov 03 01:36:07.718236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.718185    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" event={"ID":"1f6cea55-70d2-4197-8d23-72c494a1e4a1","Type":"ContainerDied","Data":"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"}
Nov 03 01:36:07.718377 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.718239    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn" event={"ID":"1f6cea55-70d2-4197-8d23-72c494a1e4a1","Type":"ContainerDied","Data":"86c739022889654563a094ee2e0b76ecb3081272da7aa50c1d8b6ed339989941"}
Nov 03 01:36:07.718377 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.718259    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"
Nov 03 01:36:07.728868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.728767    4531 scope.go:122] "RemoveContainer" containerID="20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"
Nov 03 01:36:07.729228 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:07.729148    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6\": container with ID starting with 20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6 not found: ID does not exist" containerID="20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"
Nov 03 01:36:07.729330 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.729239    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6"} err="failed to get container status \"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6\": rpc error: code = NotFound desc = could not find container \"20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6\": container with ID starting with 20086cdcfbead45d610ebd12d2fb15aee169a5338131ecc5c1165f16e1d7f2c6 not found: ID does not exist"
Nov 03 01:36:07.729330 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.729264    4531 scope.go:122] "RemoveContainer" containerID="bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"
Nov 03 01:36:07.741004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.740976    4531 scope.go:122] "RemoveContainer" containerID="bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"
Nov 03 01:36:07.741622 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:07.741314    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb\": container with ID starting with bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb not found: ID does not exist" containerID="bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"
Nov 03 01:36:07.741622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.741351    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb"} err="failed to get container status \"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb\": rpc error: code = NotFound desc = could not find container \"bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb\": container with ID starting with bce6f917cdcfe75adebab0115d2ead333614236a18c0a58c5182c015678167cb not found: ID does not exist"
Nov 03 01:36:07.741622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.741372    4531 scope.go:122] "RemoveContainer" containerID="636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"
Nov 03 01:36:07.752410 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.752386    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:36:07.754358 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.754337    4531 scope.go:122] "RemoveContainer" containerID="636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"
Nov 03 01:36:07.755017 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:07.754950    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c\": container with ID starting with 636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c not found: ID does not exist" containerID="636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"
Nov 03 01:36:07.755017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.754986    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c"} err="failed to get container status \"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c\": rpc error: code = NotFound desc = could not find container \"636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c\": container with ID starting with 636fffa0d212d6e842053f4876a331c021e3c71c07b45619b8391c5f9984a45c not found: ID does not exist"
Nov 03 01:36:07.759291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.759267    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-njbxm"]
Nov 03 01:36:07.771475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.771448    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:36:07.776251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.776230    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-n7chn"]
Nov 03 01:36:07.784949 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.784892    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:36:07.790337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:07.790311    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-737bf2d8-openshift-default-56dc96b957-rhc89"]
Nov 03 01:36:08.904943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:08.904912    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" path="/var/lib/kubelet/pods/1660361f-e8b7-43df-bd7b-3950dfa5e88d/volumes"
Nov 03 01:36:08.905423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:08.905403    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" path="/var/lib/kubelet/pods/1f6cea55-70d2-4197-8d23-72c494a1e4a1/volumes"
Nov 03 01:36:08.905994 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:08.905972    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" path="/var/lib/kubelet/pods/f784a2e6-e82e-4c45-a36c-6c1d3f10b640/volumes"
Nov 03 01:36:09.578933 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.578900    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:36:09.613109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.613082    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:36:09.670366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.670327    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\") pod \"35e2f601-0529-4a46-8566-2fce413f8879\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") "
Nov 03 01:36:09.670574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.670477    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\" (UniqueName: \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\") pod \"35e2f601-0529-4a46-8566-2fce413f8879\" (UID: \"35e2f601-0529-4a46-8566-2fce413f8879\") "
Nov 03 01:36:09.670574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.670563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\") pod \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") "
Nov 03 01:36:09.670758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.670735    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\" (UniqueName: \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\") pod \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\" (UID: \"7f72c0ac-adef-4be1-bac6-3078aebbb44d\") "
Nov 03 01:36:09.671017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.670993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume" pod "7f72c0ac-adef-4be1-bac6-3078aebbb44d" (UID: "7f72c0ac-adef-4be1-bac6-3078aebbb44d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:09.671347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.671318    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume" pod "35e2f601-0529-4a46-8566-2fce413f8879" (UID: "35e2f601-0529-4a46-8566-2fce413f8879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:09.671588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.671566    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72c0ac-adef-4be1-bac6-3078aebbb44d-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:09.673556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.673529    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc" pod "7f72c0ac-adef-4be1-bac6-3078aebbb44d" (UID: "7f72c0ac-adef-4be1-bac6-3078aebbb44d"). InnerVolumeSpecName "kube-api-access-p9svc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:09.674123 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.674099    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb" pod "35e2f601-0529-4a46-8566-2fce413f8879" (UID: "35e2f601-0529-4a46-8566-2fce413f8879"). InnerVolumeSpecName "kube-api-access-l9hhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:09.761822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.760716    4531 generic.go:356] "Generic (PLEG): container finished" podID="35e2f601-0529-4a46-8566-2fce413f8879" containerID="54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b" exitCode=0
Nov 03 01:36:09.761822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.761054    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" event={"ID":"35e2f601-0529-4a46-8566-2fce413f8879","Type":"ContainerDied","Data":"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"}
Nov 03 01:36:09.761822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.761111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh" event={"ID":"35e2f601-0529-4a46-8566-2fce413f8879","Type":"ContainerDied","Data":"98f827daff4659f391825adace0715bfbda6dd4fda29fce34692842556015b6c"}
Nov 03 01:36:09.761822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.761140    4531 scope.go:122] "RemoveContainer" containerID="54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"
Nov 03 01:36:09.761822 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.761283    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"
Nov 03 01:36:09.780890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.780848    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9svc\" (UniqueName: \"kubernetes.io/projected/7f72c0ac-adef-4be1-bac6-3078aebbb44d-kube-api-access-p9svc\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:09.780890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.780880    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35e2f601-0529-4a46-8566-2fce413f8879-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:09.780890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.780894    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hhb\" (UniqueName: \"kubernetes.io/projected/35e2f601-0529-4a46-8566-2fce413f8879-kube-api-access-l9hhb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:09.782381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.782351    4531 generic.go:356] "Generic (PLEG): container finished" podID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerID="6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db" exitCode=0
Nov 03 01:36:09.782538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.782515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" event={"ID":"7f72c0ac-adef-4be1-bac6-3078aebbb44d","Type":"ContainerDied","Data":"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"}
Nov 03 01:36:09.782671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.782585    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj" event={"ID":"7f72c0ac-adef-4be1-bac6-3078aebbb44d","Type":"ContainerDied","Data":"1d43635b53d91537c66ec689134633c3334d39a90d1c978da70f4e0213be5f27"}
Nov 03 01:36:09.782737 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.782690    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"
Nov 03 01:36:09.786658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.786627    4531 scope.go:122] "RemoveContainer" containerID="54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"
Nov 03 01:36:09.786965 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:09.786944    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b\": container with ID starting with 54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b not found: ID does not exist" containerID="54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"
Nov 03 01:36:09.787037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.786978    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b"} err="failed to get container status \"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b\": rpc error: code = NotFound desc = could not find container \"54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b\": container with ID starting with 54fb9dc15bea911b65cf1131f031a1d5c46570e2fd0f8f1d1c76d8bdd852c43b not found: ID does not exist"
Nov 03 01:36:09.787037 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.787002    4531 scope.go:122] "RemoveContainer" containerID="6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"
Nov 03 01:36:09.797783 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.797766    4531 scope.go:122] "RemoveContainer" containerID="6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"
Nov 03 01:36:09.798058 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:09.798039    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db\": container with ID starting with 6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db not found: ID does not exist" containerID="6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"
Nov 03 01:36:09.798110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.798066    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db"} err="failed to get container status \"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db\": rpc error: code = NotFound desc = could not find container \"6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db\": container with ID starting with 6180b6367e27cbc9f25d7b8cb97a72b0ff40968ea4f2a907382060049771b3db not found: ID does not exist"
Nov 03 01:36:09.806263 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.806226    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:36:09.811566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.811541    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6fec8a-c89f98cff-l8txh"]
Nov 03 01:36:09.819588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.819559    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:36:09.823653 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:09.823627    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6fec8a-55b98d9959-lspsj"]
Nov 03 01:36:09.879390 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:09.879355    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:36:10.904972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:10.904936    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e2f601-0529-4a46-8566-2fce413f8879" path="/var/lib/kubelet/pods/35e2f601-0529-4a46-8566-2fce413f8879/volumes"
Nov 03 01:36:10.905605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:10.905583    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" path="/var/lib/kubelet/pods/7f72c0ac-adef-4be1-bac6-3078aebbb44d/volumes"
Nov 03 01:36:22.022649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.022607    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:36:22.023149 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.022899    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" podUID="a9682bde-7848-468a-9720-777f57154a68" containerName="maas-api" containerID="cri-o://ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3" gracePeriod=30
Nov 03 01:36:22.603443 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.603416    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:36:22.764286 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.764173    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\") pod \"a9682bde-7848-468a-9720-777f57154a68\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") "
Nov 03 01:36:22.764473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.764298    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\") pod \"a9682bde-7848-468a-9720-777f57154a68\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") "
Nov 03 01:36:22.764473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.764340    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\") pod \"a9682bde-7848-468a-9720-777f57154a68\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") "
Nov 03 01:36:22.764473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.764423    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\" (UniqueName: \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\") pod \"a9682bde-7848-468a-9720-777f57154a68\" (UID: \"a9682bde-7848-468a-9720-777f57154a68\") "
Nov 03 01:36:22.764754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.764721    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca" pod "a9682bde-7848-468a-9720-777f57154a68" (UID: "a9682bde-7848-468a-9720-777f57154a68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:22.765357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.765331    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9682bde-7848-468a-9720-777f57154a68-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:22.766575 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.766548    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs" pod "a9682bde-7848-468a-9720-777f57154a68" (UID: "a9682bde-7848-468a-9720-777f57154a68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:22.766694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.766656    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls" pod "a9682bde-7848-468a-9720-777f57154a68" (UID: "a9682bde-7848-468a-9720-777f57154a68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:22.766869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.766853    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq" pod "a9682bde-7848-468a-9720-777f57154a68" (UID: "a9682bde-7848-468a-9720-777f57154a68"). InnerVolumeSpecName "kube-api-access-qnhtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:22.866732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.866691    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:22.866732 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.866729    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnhtq\" (UniqueName: \"kubernetes.io/projected/a9682bde-7848-468a-9720-777f57154a68-kube-api-access-qnhtq\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:22.866943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.866745    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9682bde-7848-468a-9720-777f57154a68-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:22.891769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.891738    4531 generic.go:356] "Generic (PLEG): container finished" podID="a9682bde-7848-468a-9720-777f57154a68" containerID="ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3" exitCode=0
Nov 03 01:36:22.891941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.891785    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"
Nov 03 01:36:22.891941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.891828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" event={"ID":"a9682bde-7848-468a-9720-777f57154a68","Type":"ContainerDied","Data":"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"}
Nov 03 01:36:22.891941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.891888    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" event={"ID":"a9682bde-7848-468a-9720-777f57154a68","Type":"ContainerDied","Data":"8bb6bce8f0bff013ac1529e930a47c5c4497d7cfe74a9b400368229f61d4031a"}
Nov 03 01:36:22.891941 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.891910    4531 scope.go:122] "RemoveContainer" containerID="ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"
Nov 03 01:36:22.905202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.905184    4531 scope.go:122] "RemoveContainer" containerID="ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"
Nov 03 01:36:22.905532 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:22.905498    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3\": container with ID starting with ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3 not found: ID does not exist" containerID="ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"
Nov 03 01:36:22.905647 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.905539    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3"} err="failed to get container status \"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3\": rpc error: code = NotFound desc = could not find container \"ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3\": container with ID starting with ea23741b1663b201330f8d78de781fe3f92154f53605f481a3e51cf1f31a14f3 not found: ID does not exist"
Nov 03 01:36:22.918641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.918617    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:36:22.922549 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:22.922526    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5"]
Nov 03 01:36:24.907851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:24.907805    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9682bde-7848-468a-9720-777f57154a68" path="/var/lib/kubelet/pods/a9682bde-7848-468a-9720-777f57154a68/volumes"
Nov 03 01:36:25.587530 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:25.587481    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dab6a-849b67876d-z8fm5" podUID="a9682bde-7848-468a-9720-777f57154a68" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.173:8443/health\": context deadline exceeded"
Nov 03 01:36:31.859773 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.859734    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861220    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861246    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861269    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861279    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861294    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861302    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861327    4531 cpu_manager.go:414] "containerMap: removing container" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861334    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861354    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a9682bde-7848-468a-9720-777f57154a68" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861361    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9682bde-7848-468a-9720-777f57154a68" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861373    4531 cpu_manager.go:414] "containerMap: removing container" podUID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861380    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861395    4531 cpu_manager.go:414] "containerMap: removing container" podUID="eb053be2-0283-41dc-aab5-8bc980205edb" containerName="payload-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861402    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb053be2-0283-41dc-aab5-8bc980205edb" containerName="payload-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861417    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861424    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861451    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3a53354e-580b-4411-98d1-46e59aa3ab56" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861458    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a53354e-580b-4411-98d1-46e59aa3ab56" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861614    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9682bde-7848-468a-9720-777f57154a68" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861630    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ff3ca3-1d34-4469-ae60-1486f13d2f94" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861644    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb053be2-0283-41dc-aab5-8bc980205edb" containerName="payload-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861657    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6cea55-70d2-4197-8d23-72c494a1e4a1" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861668    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a53354e-580b-4411-98d1-46e59aa3ab56" containerName="maas-api"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861682    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1660361f-e8b7-43df-bd7b-3950dfa5e88d" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861693    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f72c0ac-adef-4be1-bac6-3078aebbb44d" containerName="payload-pre-processing"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861702    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f784a2e6-e82e-4c45-a36c-6c1d3f10b640" containerName="istio-proxy"
Nov 03 01:36:31.862657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.861714    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e2f601-0529-4a46-8566-2fce413f8879" containerName="payload-processing"
Nov 03 01:36:31.884104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.879452    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:36:31.884104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.879620    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929422 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929386    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929440    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929596 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929472    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929691 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929668    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929708    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hgh\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.929954 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.929932    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.930018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.930003    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.930137 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.930115    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:31.975727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:31.975694    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-531447-openshift-default-dockercfg-z7zxz\""
Nov 03 01:36:32.031718 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031679    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.031910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031732    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.031910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031770    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hgh\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.031910 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.031982    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032106 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032116    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032308 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032306    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032488 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032582 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032505    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032761 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032743    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.032861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.032840    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.034564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.034532    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.034695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.034673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.047525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.047501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.047730 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.047709    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hgh\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.199400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.199312    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:32.527922 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.527892    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:36:32.530696 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:36:32.530659    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f2ca1b_944c_4be3_9532_0a9c0120a625.slice/crio-5c8161301e6fd9bea87b9db41fc21636e06299cbe6245e3e8df0ba310e265c12 WatchSource:0}: Error finding container 5c8161301e6fd9bea87b9db41fc21636e06299cbe6245e3e8df0ba310e265c12: Status 404 returned error can't find the container with id 5c8161301e6fd9bea87b9db41fc21636e06299cbe6245e3e8df0ba310e265c12
Nov 03 01:36:32.534282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.534243    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:32.534382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.534334    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:32.534382 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:32.534376    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:33.033449 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.033405    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" event={"ID":"57f2ca1b-944c-4be3-9532-0a9c0120a625","Type":"ContainerStarted","Data":"cab0cab3920a94d57233759b2bc7907d73362553312b49241ccf70a905b9ef99"}
Nov 03 01:36:33.033962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.033484    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" event={"ID":"57f2ca1b-944c-4be3-9532-0a9c0120a625","Type":"ContainerStarted","Data":"5c8161301e6fd9bea87b9db41fc21636e06299cbe6245e3e8df0ba310e265c12"}
Nov 03 01:36:33.059565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.059512    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podStartSLOduration=2.059497727 podStartE2EDuration="2.059497727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:33.052521969 +0000 UTC m=+3338.266265641" watchObservedRunningTime="2026-11-03 01:36:33.059497727 +0000 UTC m=+3338.273241400"
Nov 03 01:36:33.200342 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.200310    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:33.202005 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.201979    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:33.202138 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:33.202026    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:34.200659 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:34.200612    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:34.201064 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:34.200686    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:35.223197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:35.222627    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:35.223197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:35.222707    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:36.202794 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:36.202746    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:36.203943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:36.203909    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:36.473375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:36.473246    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 01:36:36.497188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:36.492503    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:36.587519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:36.587461    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:37.009683 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:37.009635    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:37.294117 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:37.268719    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:37.299188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:37.294273    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:37.862185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:37.861896    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:36:38.201088 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:38.200999    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:38.201398 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:38.201354    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:38.312112 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:38.311383    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:39.200721 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:39.200496    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Nov 03 01:36:39.200721 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:39.200555    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Nov 03 01:36:39.886497 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:36:39.886454    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:36:40.205058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:40.204971    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:40.205559 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:40.205374    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:40.206208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:40.206171    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:36:42.862914 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:42.862851    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing" containerID="cri-o://5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb" gracePeriod=25
Nov 03 01:36:46.877735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.877695    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:36:46.883885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.883855    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.892131 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.892099    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:36:46.918758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.918719    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.918951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.918767    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919022 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.918931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919085 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919007    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvr2\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919085 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919077    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919217 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919205    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919257    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919367 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919350    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:46.919506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:46.919490    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021013    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021056 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021055    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021129    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvr2\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021154    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021242    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021360 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021381    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021491    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021525    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021688    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.021964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.021939    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.022212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.022195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.022359 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.022333    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.023635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.023606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.023756 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.023705    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.029642 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.029618    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.029900 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.029885    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvr2\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.216570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.216473    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:47.515825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.515792    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:36:47.516512 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:36:47.516484    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f7cf05_97d2_4b96_a59c_b132064a34c8.slice/crio-7e15db6c723687e962d9c2860f4f9ce9909663f38d4781616d47ad90cdf03b6e WatchSource:0}: Error finding container 7e15db6c723687e962d9c2860f4f9ce9909663f38d4781616d47ad90cdf03b6e: Status 404 returned error can't find the container with id 7e15db6c723687e962d9c2860f4f9ce9909663f38d4781616d47ad90cdf03b6e
Nov 03 01:36:47.520854 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.520810    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:47.520990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.520900    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:47.520990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.520947    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:36:47.631188 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:47.630331    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:36:48.418117 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:48.418081    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" event={"ID":"d8f7cf05-97d2-4b96-a59c-b132064a34c8","Type":"ContainerStarted","Data":"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"}
Nov 03 01:36:48.418580 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:48.418144    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" event={"ID":"d8f7cf05-97d2-4b96-a59c-b132064a34c8","Type":"ContainerStarted","Data":"7e15db6c723687e962d9c2860f4f9ce9909663f38d4781616d47ad90cdf03b6e"}
Nov 03 01:36:49.216888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:49.216849    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:49.218298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:49.218268    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:49.218406 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:49.218317    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:50.217587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:50.217546    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:50.217997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:50.217605    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:51.216933 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:51.216898    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:51.217095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:51.216956    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:52.217489 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:52.217450    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:52.217896 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:52.217510    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:53.217475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:53.217439    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:53.217649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:53.217498    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:54.217041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:54.217005    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Nov 03 01:36:54.217274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:54.217091    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Nov 03 01:36:55.237186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:55.232537    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:55.237186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:55.233865    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:55.246292 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:55.243523    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:36:55.294219 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:55.294136    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podStartSLOduration=9.294116804 podStartE2EDuration="9.294116804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:48.441413195 +0000 UTC m=+3353.655156870" watchObservedRunningTime="2026-11-03 01:36:55.294116804 +0000 UTC m=+3360.507860477"
Nov 03 01:36:57.600270 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:36:57.600230    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:37:01.898021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:01.897976    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:37:01.903340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:01.903313    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:01.917801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:01.917730    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:37:02.013957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.013921    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.013991    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014124    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014403 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014377    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014473 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014533 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kq4n\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014582 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014539    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014651 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014633    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.014906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.014882    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.116583 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116546    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.116779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116644    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.116779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116689    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.116779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116731    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.116779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116765    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116841    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116869    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kq4n\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.116936    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117323 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.117297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.117475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117612 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.117391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117683 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.117321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.117841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.117782    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.119366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.119341    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.119563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.119539    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.124446 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.124426    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.124573 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.124473    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kq4n\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\") pod \"e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") " pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.225059 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.224976    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:02.528605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.528566    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:37:02.529692 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:37:02.529663    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d52c5b6_667d_4d5d_a11c_eccd90451b47.slice/crio-9ed4c06e93f4a29237a33c5b26e22b28711ffe3ef0449fc290ec54f03fa04d7f WatchSource:0}: Error finding container 9ed4c06e93f4a29237a33c5b26e22b28711ffe3ef0449fc290ec54f03fa04d7f: Status 404 returned error can't find the container with id 9ed4c06e93f4a29237a33c5b26e22b28711ffe3ef0449fc290ec54f03fa04d7f
Nov 03 01:37:02.533243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.533199    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:37:02.533364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.533291    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:37:02.533364 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.533332    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:37:02.591440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:02.591403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" event={"ID":"0d52c5b6-667d-4d5d-a11c-eccd90451b47","Type":"ContainerStarted","Data":"9ed4c06e93f4a29237a33c5b26e22b28711ffe3ef0449fc290ec54f03fa04d7f"}
Nov 03 01:37:03.217458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.217428    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:37:03.368182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.366669    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\") pod \"1b082ef0-f9b7-4096-a60c-0da0636c1846\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") "
Nov 03 01:37:03.368182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.366978    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\" (UniqueName: \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\") pod \"1b082ef0-f9b7-4096-a60c-0da0636c1846\" (UID: \"1b082ef0-f9b7-4096-a60c-0da0636c1846\") "
Nov 03 01:37:03.377678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.377621    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume" pod "1b082ef0-f9b7-4096-a60c-0da0636c1846" (UID: "1b082ef0-f9b7-4096-a60c-0da0636c1846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:03.403580 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.400118    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p" pod "1b082ef0-f9b7-4096-a60c-0da0636c1846" (UID: "1b082ef0-f9b7-4096-a60c-0da0636c1846"). InnerVolumeSpecName "kube-api-access-lqc4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:03.469092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.469050    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqc4p\" (UniqueName: \"kubernetes.io/projected/1b082ef0-f9b7-4096-a60c-0da0636c1846-kube-api-access-lqc4p\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:03.469092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.469091    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b082ef0-f9b7-4096-a60c-0da0636c1846-plugins-config-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:03.599967 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.599924    4531 generic.go:356] "Generic (PLEG): container finished" podID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerID="5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb" exitCode=0
Nov 03 01:37:03.600186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.599975    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"
Nov 03 01:37:03.600186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.600028    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" event={"ID":"1b082ef0-f9b7-4096-a60c-0da0636c1846","Type":"ContainerDied","Data":"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"}
Nov 03 01:37:03.600186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.600095    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj" event={"ID":"1b082ef0-f9b7-4096-a60c-0da0636c1846","Type":"ContainerDied","Data":"113c0c560c461322b6ab83de2dd180c961355fb85a02d1ff075dbabccc54302e"}
Nov 03 01:37:03.600186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.600119    4531 scope.go:122] "RemoveContainer" containerID="5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"
Nov 03 01:37:03.602302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.602275    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" event={"ID":"0d52c5b6-667d-4d5d-a11c-eccd90451b47","Type":"ContainerStarted","Data":"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"}
Nov 03 01:37:03.612134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.612116    4531 scope.go:122] "RemoveContainer" containerID="5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"
Nov 03 01:37:03.612432 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:37:03.612411    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb\": container with ID starting with 5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb not found: ID does not exist" containerID="5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"
Nov 03 01:37:03.612505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.612438    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb"} err="failed to get container status \"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb\": rpc error: code = NotFound desc = could not find container \"5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb\": container with ID starting with 5867bf64da43f63f148ada6a51ad1bb6adaac4017791b8f8c06ccfb9f9838feb not found: ID does not exist"
Nov 03 01:37:03.623114 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.623067    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podStartSLOduration=2.6230552190000003 podStartE2EDuration="2.623055219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:03.622615316 +0000 UTC m=+3368.836358987" watchObservedRunningTime="2026-11-03 01:37:03.623055219 +0000 UTC m=+3368.836798892"
Nov 03 01:37:03.663569 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.663466    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:37:03.667937 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:03.667901    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-be59df-bb748fd99-jzkdj"]
Nov 03 01:37:04.225508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:04.225472    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:04.232183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:04.231267    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:04.232183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:04.231341    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:04.907824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:04.907786    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" path="/var/lib/kubelet/pods/1b082ef0-f9b7-4096-a60c-0da0636c1846/volumes"
Nov 03 01:37:05.225769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:05.225681    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:05.225769 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:05.225744    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:06.230187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:06.226312    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:06.230187 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:06.226390    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:07.226009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:07.225921    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:07.226009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:07.225974    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:08.226344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:08.226307    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:08.226940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:08.226907    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:09.226285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:09.226245    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 01:37:09.226461 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:09.226305    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 01:37:09.882772 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:37:09.882731    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:37:10.257841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:10.257540    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:10.258834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:10.258374    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:10.259614 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:10.259596    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:37:11.364752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:11.364686    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:37:11.365492 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:11.364928    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" podUID="d03d2a92-b898-466b-a679-b94d45070b0d" containerName="maas-api" containerID="cri-o://38e810e58be171db4108c8285d688d1a1d82ebb17b0c84ad3005524f9cf43871" gracePeriod=30
Nov 03 01:37:11.739487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:11.738083    4531 generic.go:356] "Generic (PLEG): container finished" podID="d03d2a92-b898-466b-a679-b94d45070b0d" containerID="38e810e58be171db4108c8285d688d1a1d82ebb17b0c84ad3005524f9cf43871" exitCode=0
Nov 03 01:37:11.739487 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:11.739357    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" event={"ID":"d03d2a92-b898-466b-a679-b94d45070b0d","Type":"ContainerDied","Data":"38e810e58be171db4108c8285d688d1a1d82ebb17b0c84ad3005524f9cf43871"}
Nov 03 01:37:12.015567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.015535    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:37:12.185368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.185331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\" (UniqueName: \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\") pod \"d03d2a92-b898-466b-a679-b94d45070b0d\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") "
Nov 03 01:37:12.185564 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.185503    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\") pod \"d03d2a92-b898-466b-a679-b94d45070b0d\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") "
Nov 03 01:37:12.185639 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.185606    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") pod \"d03d2a92-b898-466b-a679-b94d45070b0d\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") "
Nov 03 01:37:12.185726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.185704    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\") pod \"d03d2a92-b898-466b-a679-b94d45070b0d\" (UID: \"d03d2a92-b898-466b-a679-b94d45070b0d\") "
Nov 03 01:37:12.185970 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.185936    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca" pod "d03d2a92-b898-466b-a679-b94d45070b0d" (UID: "d03d2a92-b898-466b-a679-b94d45070b0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:12.186455 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.186430    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03d2a92-b898-466b-a679-b94d45070b0d-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:12.187767 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.187739    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls" pod "d03d2a92-b898-466b-a679-b94d45070b0d" (UID: "d03d2a92-b898-466b-a679-b94d45070b0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:12.187904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.187887    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw" pod "d03d2a92-b898-466b-a679-b94d45070b0d" (UID: "d03d2a92-b898-466b-a679-b94d45070b0d"). InnerVolumeSpecName "kube-api-access-cv7bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:12.187949 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.187907    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs" pod "d03d2a92-b898-466b-a679-b94d45070b0d" (UID: "d03d2a92-b898-466b-a679-b94d45070b0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:12.287592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.287497    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:12.287592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.287529    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03d2a92-b898-466b-a679-b94d45070b0d-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:12.287592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.287543    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv7bw\" (UniqueName: \"kubernetes.io/projected/d03d2a92-b898-466b-a679-b94d45070b0d-kube-api-access-cv7bw\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:12.749832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.749797    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"
Nov 03 01:37:12.749832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.749821    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb" event={"ID":"d03d2a92-b898-466b-a679-b94d45070b0d","Type":"ContainerDied","Data":"c3ccfc368fd0b9471c2c35ca921cfd5c95c31b51ecaa04725e053a4a787071c7"}
Nov 03 01:37:12.750460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.749898    4531 scope.go:122] "RemoveContainer" containerID="38e810e58be171db4108c8285d688d1a1d82ebb17b0c84ad3005524f9cf43871"
Nov 03 01:37:12.781631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.781598    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:37:12.785825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.785793    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-50e623-6949cf45c-v28rb"]
Nov 03 01:37:12.905523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:12.905491    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03d2a92-b898-466b-a679-b94d45070b0d" path="/var/lib/kubelet/pods/d03d2a92-b898-466b-a679-b94d45070b0d/volumes"
Nov 03 01:37:15.551016 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:15.550978    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:37:15.552523 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:15.552023    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy" containerID="cri-o://262b842b3a5d644c37e0197450cad6c918999286bec13131f1cc7eed52bc1aee" gracePeriod=30
Nov 03 01:37:20.870657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:20.870559    4531 generic.go:356] "Generic (PLEG): container finished" podID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerID="262b842b3a5d644c37e0197450cad6c918999286bec13131f1cc7eed52bc1aee" exitCode=0
Nov 03 01:37:20.871145 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:20.870812    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" event={"ID":"e70ae258-f4f8-41a2-988b-74f9e7d75fc3","Type":"ContainerDied","Data":"262b842b3a5d644c37e0197450cad6c918999286bec13131f1cc7eed52bc1aee"}
Nov 03 01:37:20.952387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:20.952359    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:37:21.110483 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110440    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.110715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110523    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.110715 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110646    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.110878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110723    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.110878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110776    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:37:21.110878 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.111087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.110896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.111087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111011    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.111087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111064    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.111287 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\") pod \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\" (UID: \"e70ae258-f4f8-41a2-988b-74f9e7d75fc3\") "
Nov 03 01:37:21.111707 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111677    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:37:21.111893 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111873    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:37:21.112017 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.111991    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:21.112175 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:37:21.112587 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112564    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.112695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112592    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.112695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112607    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.112695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112621    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.112695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.112634    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.114861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.114832    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:37:21.115645 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.115619    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:37:21.115968 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.115933    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "kube-api-access-7pmzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:21.118598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.118569    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token" pod "e70ae258-f4f8-41a2-988b-74f9e7d75fc3" (UID: "e70ae258-f4f8-41a2-988b-74f9e7d75fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:21.213975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.213936    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.213975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.213971    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmzb\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-kube-api-access-7pmzb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.214238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.213986    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.214238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.213999    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e70ae258-f4f8-41a2-988b-74f9e7d75fc3-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:21.906182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.903646    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg" event={"ID":"e70ae258-f4f8-41a2-988b-74f9e7d75fc3","Type":"ContainerDied","Data":"ddd75ed7d7d369bf5e37d7e693c2ef3edd940799cf4a4d8adf8df3be0c308fa7"}
Nov 03 01:37:21.906182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.903727    4531 scope.go:122] "RemoveContainer" containerID="262b842b3a5d644c37e0197450cad6c918999286bec13131f1cc7eed52bc1aee"
Nov 03 01:37:21.906182 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.903928    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"
Nov 03 01:37:21.955607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.955568    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:37:21.956863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:21.956839    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-50e623-openshift-default-6c86dc794f-4mplg"]
Nov 03 01:37:22.905191 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:22.905134    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" path="/var/lib/kubelet/pods/e70ae258-f4f8-41a2-988b-74f9e7d75fc3/volumes"
Nov 03 01:37:25.001888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.001844    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:37:25.002426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.002097    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" podUID="d3e0b936-03da-4006-9d61-75118e093a9d" containerName="maas-api" containerID="cri-o://de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c" gracePeriod=30
Nov 03 01:37:25.628689 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.628659    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:37:25.649047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.649014    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\") pod \"d3e0b936-03da-4006-9d61-75118e093a9d\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") "
Nov 03 01:37:25.649329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.649310    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\") pod \"d3e0b936-03da-4006-9d61-75118e093a9d\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") "
Nov 03 01:37:25.649858 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.649836    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\" (UniqueName: \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\") pod \"d3e0b936-03da-4006-9d61-75118e093a9d\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") "
Nov 03 01:37:25.659370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.658683    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\") pod \"d3e0b936-03da-4006-9d61-75118e093a9d\" (UID: \"d3e0b936-03da-4006-9d61-75118e093a9d\") "
Nov 03 01:37:25.659370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.652460    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs" pod "d3e0b936-03da-4006-9d61-75118e093a9d" (UID: "d3e0b936-03da-4006-9d61-75118e093a9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:25.659370 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.658448    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls" pod "d3e0b936-03da-4006-9d61-75118e093a9d" (UID: "d3e0b936-03da-4006-9d61-75118e093a9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:25.659612 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.659474    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:25.659612 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.659497    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e0b936-03da-4006-9d61-75118e093a9d-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:25.659946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.659895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f" pod "d3e0b936-03da-4006-9d61-75118e093a9d" (UID: "d3e0b936-03da-4006-9d61-75118e093a9d"). InnerVolumeSpecName "kube-api-access-hw65f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:25.659946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.659912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca" pod "d3e0b936-03da-4006-9d61-75118e093a9d" (UID: "d3e0b936-03da-4006-9d61-75118e093a9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:25.761904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.761862    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw65f\" (UniqueName: \"kubernetes.io/projected/d3e0b936-03da-4006-9d61-75118e093a9d-kube-api-access-hw65f\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:25.761904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.761906    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e0b936-03da-4006-9d61-75118e093a9d-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:37:25.937629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.937526    4531 generic.go:356] "Generic (PLEG): container finished" podID="d3e0b936-03da-4006-9d61-75118e093a9d" containerID="de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c" exitCode=0
Nov 03 01:37:25.937629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.937583    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"
Nov 03 01:37:25.937849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.937616    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" event={"ID":"d3e0b936-03da-4006-9d61-75118e093a9d","Type":"ContainerDied","Data":"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"}
Nov 03 01:37:25.937849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.937684    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn" event={"ID":"d3e0b936-03da-4006-9d61-75118e093a9d","Type":"ContainerDied","Data":"e1bc4cf17d9b84fe287463fea7d8cd82b960ea29c7ea46e8b1603bbdd453a0e3"}
Nov 03 01:37:25.937849 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.937706    4531 scope.go:122] "RemoveContainer" containerID="de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"
Nov 03 01:37:25.951014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.950596    4531 scope.go:122] "RemoveContainer" containerID="de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"
Nov 03 01:37:25.951014 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:37:25.950883    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c\": container with ID starting with de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c not found: ID does not exist" containerID="de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"
Nov 03 01:37:25.951014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.950913    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c"} err="failed to get container status \"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c\": rpc error: code = NotFound desc = could not find container \"de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c\": container with ID starting with de8f0b13cf69758a82435fcdb6eae3d5b92c4c83455f96faf92e2b034507978c not found: ID does not exist"
Nov 03 01:37:25.966929 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.966901    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:37:25.971658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:25.971629    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-020af9-5c87596c65-842jn"]
Nov 03 01:37:26.904102 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:26.904070    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e0b936-03da-4006-9d61-75118e093a9d" path="/var/lib/kubelet/pods/d3e0b936-03da-4006-9d61-75118e093a9d/volumes"
Nov 03 01:37:39.882018 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:37:39.881978    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:37:44.887474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:44.887438    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 01:37:44.914233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:44.914201    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 01:37:59.914039 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:59.913998    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main" containerID="cri-o://725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c" gracePeriod=45
Nov 03 01:37:59.946736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:37:59.946683    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" containerID="cri-o://2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c" gracePeriod=45
Nov 03 01:38:00.103703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.103646    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Nov 03 01:38:00.281543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.281507    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:38:00.295692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.295648    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:38:00.400740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.400707    4531 generic.go:356] "Generic (PLEG): container finished" podID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerID="725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c" exitCode=0
Nov 03 01:38:00.400931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.400749    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"
Nov 03 01:38:00.400931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.400789    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" event={"ID":"675f003d-3ddc-4e7f-9743-00e7119b1ab5","Type":"ContainerDied","Data":"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"}
Nov 03 01:38:00.400931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.400831    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc" event={"ID":"675f003d-3ddc-4e7f-9743-00e7119b1ab5","Type":"ContainerDied","Data":"408ce894a580cac0c4f958c37ce3a5683149fd606500e16f525cd7a038b4398c"}
Nov 03 01:38:00.400931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.400851    4531 scope.go:122] "RemoveContainer" containerID="725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"
Nov 03 01:38:00.402372 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.402351    4531 generic.go:356] "Generic (PLEG): container finished" podID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerID="2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c" exitCode=0
Nov 03 01:38:00.402493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.402408    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"
Nov 03 01:38:00.402493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.402444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" event={"ID":"b9ba8ade-6968-438c-b4f0-3ef913b858ad","Type":"ContainerDied","Data":"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"}
Nov 03 01:38:00.402493 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.402489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr" event={"ID":"b9ba8ade-6968-438c-b4f0-3ef913b858ad","Type":"ContainerDied","Data":"fff0723bcb88e84cf470cfe4be0e66b52076c400b968146f369ca1f032bfb4e6"}
Nov 03 01:38:00.412829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.412806    4531 scope.go:122] "RemoveContainer" containerID="725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"
Nov 03 01:38:00.413116 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:38:00.413093    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c\": container with ID starting with 725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c not found: ID does not exist" containerID="725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"
Nov 03 01:38:00.413206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.413130    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c"} err="failed to get container status \"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c\": rpc error: code = NotFound desc = could not find container \"725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c\": container with ID starting with 725a032760bd94bc6d51fc2bbc69f36e2a96dc468a41bc8d9b05fb384ec2fa2c not found: ID does not exist"
Nov 03 01:38:00.413206 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.413153    4531 scope.go:122] "RemoveContainer" containerID="2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"
Nov 03 01:38:00.423110 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.423088    4531 scope.go:122] "RemoveContainer" containerID="2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"
Nov 03 01:38:00.423443 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:38:00.423423    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c\": container with ID starting with 2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c not found: ID does not exist" containerID="2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"
Nov 03 01:38:00.423497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.423451    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c"} err="failed to get container status \"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c\": rpc error: code = NotFound desc = could not find container \"2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c\": container with ID starting with 2b68dc7c1bcfaf48cd7015b8976f7d9883a399c74ccb647e3b9e4eb2e580d69c not found: ID does not exist"
Nov 03 01:38:00.439459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439428    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.439577 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439490    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.439636 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439586    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.439694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439650    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.439694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439678    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.439813 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439725    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.439813 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439735    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.439813 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439782    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.439985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439823    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\" (UniqueName: \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.439985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439844    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.439985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439865    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\" (UniqueName: \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.439985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439914    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.439985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.439933    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.440351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440003    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\" (UniqueName: \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\") pod \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\" (UID: \"675f003d-3ddc-4e7f-9743-00e7119b1ab5\") "
Nov 03 01:38:00.440351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440058    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\") pod \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\" (UID: \"b9ba8ade-6968-438c-b4f0-3ef913b858ad\") "
Nov 03 01:38:00.440501 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440479    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.440726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440708    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.441130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440731    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.441130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440745    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.441130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440759    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.441130 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.440961    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.442432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.441049    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.443824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.442861    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:00.443824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.442949    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:00.443824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.443365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.443824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.443483    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2" pod "b9ba8ade-6968-438c-b4f0-3ef913b858ad" (UID: "b9ba8ade-6968-438c-b4f0-3ef913b858ad"). InnerVolumeSpecName "kube-api-access-zkhr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:00.444556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.444531    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:00.445716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.445686    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4" pod "675f003d-3ddc-4e7f-9743-00e7119b1ab5" (UID: "675f003d-3ddc-4e7f-9743-00e7119b1ab5"). InnerVolumeSpecName "kube-api-access-bb5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544413    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544460    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544477    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkhr2\" (UniqueName: \"kubernetes.io/projected/b9ba8ade-6968-438c-b4f0-3ef913b858ad-kube-api-access-zkhr2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544493    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544508    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb5h4\" (UniqueName: \"kubernetes.io/projected/675f003d-3ddc-4e7f-9743-00e7119b1ab5-kube-api-access-bb5h4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544522    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ba8ade-6968-438c-b4f0-3ef913b858ad-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544534    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ba8ade-6968-438c-b4f0-3ef913b858ad-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.544652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.544547    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/675f003d-3ddc-4e7f-9743-00e7119b1ab5-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:00.729272 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.729226    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 01:38:00.733334 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.733302    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/premium-w0-2f9a3a-kserve-68bd866d8-np5kr"]
Nov 03 01:38:00.741109 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.741078    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 01:38:00.744683 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.744654    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2f9a3a/distinct-2-w0-2f9a3a-kserve-5dbc8f59c6-cmbtc"]
Nov 03 01:38:00.905729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.905692    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" path="/var/lib/kubelet/pods/675f003d-3ddc-4e7f-9743-00e7119b1ab5/volumes"
Nov 03 01:38:00.906201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:00.906185    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" path="/var/lib/kubelet/pods/b9ba8ade-6968-438c-b4f0-3ef913b858ad/volumes"
Nov 03 01:38:09.878586 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:38:09.878542    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:38:28.883197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:28.872888    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 01:38:28.883197 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:28.873142    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy" containerID="cri-o://ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7" gracePeriod=30
Nov 03 01:38:34.182451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.182422    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 01:38:34.192465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192437    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192522    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192545    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192669 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192622    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:34.192723 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192687    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192754    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192783    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.192829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192808    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.193003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192965    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:34.193063 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.192993    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\") pod \"e3c40d84-4cde-4104-84c5-cb356da785e4\" (UID: \"e3c40d84-4cde-4104-84c5-cb356da785e4\") "
Nov 03 01:38:34.193063 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:34.193190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193079    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:34.193261 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193237    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:34.193620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193591    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.193620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193610    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c40d84-4cde-4104-84c5-cb356da785e4-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.193620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193620    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.193872 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193629    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.193872 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.193638    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.195048 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.195016    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:38:34.195290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.195268    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "kube-api-access-jcsz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:34.195378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.195313    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:38:34.195378 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.195333    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token" pod "e3c40d84-4cde-4104-84c5-cb356da785e4" (UID: "e3c40d84-4cde-4104-84c5-cb356da785e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:34.297722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.297580    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.297722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.297663    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsz6\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-kube-api-access-jcsz6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.297722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.297681    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.297722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.297695    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c40d84-4cde-4104-84c5-cb356da785e4-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:34.721770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.721733    4531 generic.go:356] "Generic (PLEG): container finished" podID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerID="ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7" exitCode=0
Nov 03 01:38:34.721770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.721772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" event={"ID":"e3c40d84-4cde-4104-84c5-cb356da785e4","Type":"ContainerDied","Data":"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"}
Nov 03 01:38:34.722020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.721779    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"
Nov 03 01:38:34.722020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.721810    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b" event={"ID":"e3c40d84-4cde-4104-84c5-cb356da785e4","Type":"ContainerDied","Data":"15c4bcbd9e777bdef3e6dcf7f01fa7b1009c39165a71a1811dbce2be84533532"}
Nov 03 01:38:34.722020 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.721833    4531 scope.go:122] "RemoveContainer" containerID="ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"
Nov 03 01:38:34.734538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.734508    4531 scope.go:122] "RemoveContainer" containerID="ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"
Nov 03 01:38:34.734835 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:38:34.734809    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7\": container with ID starting with ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7 not found: ID does not exist" containerID="ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"
Nov 03 01:38:34.734908 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.734850    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7"} err="failed to get container status \"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7\": rpc error: code = NotFound desc = could not find container \"ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7\": container with ID starting with ce1d2d17a687af161e10eaa470c81b37f68952c00cfd8a285608f80df30b84d7 not found: ID does not exist"
Nov 03 01:38:34.750963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.750931    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 01:38:34.756930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.756898    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2f9a3a-openshift-default-7f6f74cf97-6mh8b"]
Nov 03 01:38:34.906860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:38:34.906827    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" path="/var/lib/kubelet/pods/e3c40d84-4cde-4104-84c5-cb356da785e4/volumes"
Nov 03 01:38:39.882352 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:38:39.882263    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:39:09.923182 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:39:09.923129    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:39:39.881348 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:39:39.881309    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:39:49.527745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:49.524672    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:39:49.527745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:49.524934    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" podUID="555ec9eb-2fab-419c-86c1-0db07e23907d" containerName="maas-api" containerID="cri-o://32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6" gracePeriod=30
Nov 03 01:39:50.088371 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.088345    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.227071    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\") pod \"555ec9eb-2fab-419c-86c1-0db07e23907d\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") "
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.227141    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\") pod \"555ec9eb-2fab-419c-86c1-0db07e23907d\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") "
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.227425    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\" (UniqueName: \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\") pod \"555ec9eb-2fab-419c-86c1-0db07e23907d\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") "
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.227543    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\") pod \"555ec9eb-2fab-419c-86c1-0db07e23907d\" (UID: \"555ec9eb-2fab-419c-86c1-0db07e23907d\") "
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.227577    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca" pod "555ec9eb-2fab-419c-86c1-0db07e23907d" (UID: "555ec9eb-2fab-419c-86c1-0db07e23907d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:50.228251 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.228077    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555ec9eb-2fab-419c-86c1-0db07e23907d-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:50.230183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.230133    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk" pod "555ec9eb-2fab-419c-86c1-0db07e23907d" (UID: "555ec9eb-2fab-419c-86c1-0db07e23907d"). InnerVolumeSpecName "kube-api-access-nw5bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:50.230327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.230230    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls" pod "555ec9eb-2fab-419c-86c1-0db07e23907d" (UID: "555ec9eb-2fab-419c-86c1-0db07e23907d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:50.230499 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.230460    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs" pod "555ec9eb-2fab-419c-86c1-0db07e23907d" (UID: "555ec9eb-2fab-419c-86c1-0db07e23907d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:50.329539 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.329493    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:50.329539 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.329535    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw5bk\" (UniqueName: \"kubernetes.io/projected/555ec9eb-2fab-419c-86c1-0db07e23907d-kube-api-access-nw5bk\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:50.329539 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.329547    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555ec9eb-2fab-419c-86c1-0db07e23907d-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:50.483571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.483473    4531 generic.go:356] "Generic (PLEG): container finished" podID="555ec9eb-2fab-419c-86c1-0db07e23907d" containerID="32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6" exitCode=0
Nov 03 01:39:50.483571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.483526    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" event={"ID":"555ec9eb-2fab-419c-86c1-0db07e23907d","Type":"ContainerDied","Data":"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"}
Nov 03 01:39:50.483571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.483537    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"
Nov 03 01:39:50.483571 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.483570    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm" event={"ID":"555ec9eb-2fab-419c-86c1-0db07e23907d","Type":"ContainerDied","Data":"2778a2ca0cfc746bd0dfa6d3d9f4a7b13fab03a6ded5b93d288f0b528f19c528"}
Nov 03 01:39:50.483861 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.483591    4531 scope.go:122] "RemoveContainer" containerID="32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"
Nov 03 01:39:50.495993 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.495971    4531 scope.go:122] "RemoveContainer" containerID="32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"
Nov 03 01:39:50.496283 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:39:50.496261    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6\": container with ID starting with 32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6 not found: ID does not exist" containerID="32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"
Nov 03 01:39:50.496362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.496295    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6"} err="failed to get container status \"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6\": rpc error: code = NotFound desc = could not find container \"32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6\": container with ID starting with 32b87f299decd96e42b243a3bc116f02d050ca3f5ba00806abab4f4afd46cda6 not found: ID does not exist"
Nov 03 01:39:50.507419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.507392    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:39:50.511480 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.511459    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-027c19-f4cbcdcd6-6zlnm"]
Nov 03 01:39:50.904501 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:39:50.904466    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555ec9eb-2fab-419c-86c1-0db07e23907d" path="/var/lib/kubelet/pods/555ec9eb-2fab-419c-86c1-0db07e23907d/volumes"
Nov 03 01:40:09.877815 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:09.877721    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:40:10.064236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.064200    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:40:10.064847 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.064805    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerName="maas-api" containerID="cri-o://1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b" gracePeriod=30
Nov 03 01:40:10.370186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.366941    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.125:8443/health\": dial tcp 10.133.0.125:8443: connect: connection refused"
Nov 03 01:40:10.648475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.648445    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:40:10.689909 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.689876    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\") pod \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") "
Nov 03 01:40:10.690095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.690019    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\") pod \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") "
Nov 03 01:40:10.690095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.690040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\" (UniqueName: \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\") pod \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") "
Nov 03 01:40:10.690095 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.690081    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\") pod \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\" (UID: \"d4a6a44e-0da1-4400-80cf-62dc78e274e1\") "
Nov 03 01:40:10.690375 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.690349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca" pod "d4a6a44e-0da1-4400-80cf-62dc78e274e1" (UID: "d4a6a44e-0da1-4400-80cf-62dc78e274e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:10.690722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.690705    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4a6a44e-0da1-4400-80cf-62dc78e274e1-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:10.692567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.692536    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs" pod "d4a6a44e-0da1-4400-80cf-62dc78e274e1" (UID: "d4a6a44e-0da1-4400-80cf-62dc78e274e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:10.692680 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.692571    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n" pod "d4a6a44e-0da1-4400-80cf-62dc78e274e1" (UID: "d4a6a44e-0da1-4400-80cf-62dc78e274e1"). InnerVolumeSpecName "kube-api-access-2zz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:10.692770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.692754    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls" pod "d4a6a44e-0da1-4400-80cf-62dc78e274e1" (UID: "d4a6a44e-0da1-4400-80cf-62dc78e274e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:10.716077 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.716046    4531 generic.go:356] "Generic (PLEG): container finished" podID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerID="1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b" exitCode=0
Nov 03 01:40:10.716228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.716093    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"
Nov 03 01:40:10.716228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.716129    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" event={"ID":"d4a6a44e-0da1-4400-80cf-62dc78e274e1","Type":"ContainerDied","Data":"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"}
Nov 03 01:40:10.716228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.716209    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x" event={"ID":"d4a6a44e-0da1-4400-80cf-62dc78e274e1","Type":"ContainerDied","Data":"f018bfb63da55ae3451e990fef6dfc8a62947eeeb73438981c84ae39f0a3d3cd"}
Nov 03 01:40:10.716387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.716236    4531 scope.go:122] "RemoveContainer" containerID="1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"
Nov 03 01:40:10.726952 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.726928    4531 scope.go:122] "RemoveContainer" containerID="1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"
Nov 03 01:40:10.727266 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:10.727246    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b\": container with ID starting with 1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b not found: ID does not exist" containerID="1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"
Nov 03 01:40:10.727348 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.727278    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b"} err="failed to get container status \"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b\": rpc error: code = NotFound desc = could not find container \"1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b\": container with ID starting with 1a16f0d931f73f613ae82b5def1d11351466b648920db92c8d4eba9aa53a7d2b not found: ID does not exist"
Nov 03 01:40:10.741108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.741078    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:40:10.744674 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.744653    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3c5983-784c9fb758-4tp8x"]
Nov 03 01:40:10.792367 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.792336    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:10.792367 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.792364    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zz9n\" (UniqueName: \"kubernetes.io/projected/d4a6a44e-0da1-4400-80cf-62dc78e274e1-kube-api-access-2zz9n\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:10.792574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.792377    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4a6a44e-0da1-4400-80cf-62dc78e274e1-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:10.903758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:10.903679    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" path="/var/lib/kubelet/pods/d4a6a44e-0da1-4400-80cf-62dc78e274e1/volumes"
Nov 03 01:40:34.822333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:34.822295    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:40:34.822884 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:34.822572    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" podUID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" containerName="maas-api" containerID="cri-o://608835b83e85d6367411bb72387e3b2a9bc3cfdc4fabc8ceccc5cab50eef6f10" gracePeriod=30
Nov 03 01:40:34.994704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:34.994660    4531 generic.go:356] "Generic (PLEG): container finished" podID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" containerID="608835b83e85d6367411bb72387e3b2a9bc3cfdc4fabc8ceccc5cab50eef6f10" exitCode=0
Nov 03 01:40:34.994920 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:34.994779    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" event={"ID":"e018a29c-f897-4e40-a2fa-94cb5b4c56a9","Type":"ContainerDied","Data":"608835b83e85d6367411bb72387e3b2a9bc3cfdc4fabc8ceccc5cab50eef6f10"}
Nov 03 01:40:35.401819 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.401790    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:40:35.565770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.565735    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\") pod \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") "
Nov 03 01:40:35.565977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.565841    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") pod \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") "
Nov 03 01:40:35.565977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.565953    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\") pod \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") "
Nov 03 01:40:35.566195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.566172    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\" (UniqueName: \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\") pod \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\" (UID: \"e018a29c-f897-4e40-a2fa-94cb5b4c56a9\") "
Nov 03 01:40:35.566419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.566224    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca" pod "e018a29c-f897-4e40-a2fa-94cb5b4c56a9" (UID: "e018a29c-f897-4e40-a2fa-94cb5b4c56a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:35.567754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.567727    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:35.568805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.568778    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls" pod "e018a29c-f897-4e40-a2fa-94cb5b4c56a9" (UID: "e018a29c-f897-4e40-a2fa-94cb5b4c56a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:35.568805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.568798    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb" pod "e018a29c-f897-4e40-a2fa-94cb5b4c56a9" (UID: "e018a29c-f897-4e40-a2fa-94cb5b4c56a9"). InnerVolumeSpecName "kube-api-access-cjwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:35.569243 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.569219    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs" pod "e018a29c-f897-4e40-a2fa-94cb5b4c56a9" (UID: "e018a29c-f897-4e40-a2fa-94cb5b4c56a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:35.668946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.668860    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwrb\" (UniqueName: \"kubernetes.io/projected/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-kube-api-access-cjwrb\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:35.668946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.668893    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:35.668946 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:35.668916    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e018a29c-f897-4e40-a2fa-94cb5b4c56a9-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:36.007114 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.007005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r" event={"ID":"e018a29c-f897-4e40-a2fa-94cb5b4c56a9","Type":"ContainerDied","Data":"da20c4222152539bd0e1dee21dbff5d081df4da9258aabcb1ed14010778f78d3"}
Nov 03 01:40:36.007114 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.007077    4531 scope.go:122] "RemoveContainer" containerID="608835b83e85d6367411bb72387e3b2a9bc3cfdc4fabc8ceccc5cab50eef6f10"
Nov 03 01:40:36.008090 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.007295    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"
Nov 03 01:40:36.073397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.055482    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:40:36.073397 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.072078    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-531447-cd48b697d-tl55r"]
Nov 03 01:40:36.904992 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:36.904945    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" path="/var/lib/kubelet/pods/e018a29c-f897-4e40-a2fa-94cb5b4c56a9/volumes"
Nov 03 01:40:38.702185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.700747    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:40:38.702185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.701206    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" containerID="cri-o://cab0cab3920a94d57233759b2bc7907d73362553312b49241ccf70a905b9ef99" gracePeriod=30
Nov 03 01:40:38.709886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.708999    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:40:38.709886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.709224    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" containerID="cri-o://9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510" gracePeriod=30
Nov 03 01:40:38.717516 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.717489    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:40:38.717712 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:38.717692    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" containerID="cri-o://bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0" gracePeriod=30
Nov 03 01:40:39.889074 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:39.889037    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:40:40.207856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.207775    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:40:40.207856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.207844    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:40:40.236269 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.236226    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:40:40.236441 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.236307    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:40:40.260716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.260677    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:40:40.260905 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:40.260748    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:40:44.017703 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.017674    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.044825    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.044928    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.044999    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045096    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045128    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045221    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045261    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045351    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.045445    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\") pod \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\" (UID: \"0d52c5b6-667d-4d5d-a11c-eccd90451b47\") "
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.047150    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.047804 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.047578    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.048963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.048264    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.051633 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.051604    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.054046 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.054017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:44.058646 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.057772    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "kube-api-access-8kq4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.075438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.075386    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:40:44.076239 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.076210    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.078343 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.078325    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:40:44.080739 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.080711    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token" pod "0d52c5b6-667d-4d5d-a11c-eccd90451b47" (UID: "0d52c5b6-667d-4d5d-a11c-eccd90451b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.083871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.083840    4531 generic.go:356] "Generic (PLEG): container finished" podID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerID="cab0cab3920a94d57233759b2bc7907d73362553312b49241ccf70a905b9ef99" exitCode=0
Nov 03 01:40:44.083985 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.083963    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" event={"ID":"57f2ca1b-944c-4be3-9532-0a9c0120a625","Type":"ContainerDied","Data":"cab0cab3920a94d57233759b2bc7907d73362553312b49241ccf70a905b9ef99"}
Nov 03 01:40:44.085720 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.085696    4531 generic.go:356] "Generic (PLEG): container finished" podID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerID="bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0" exitCode=0
Nov 03 01:40:44.085834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.085751    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"
Nov 03 01:40:44.085834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.085779    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" event={"ID":"d8f7cf05-97d2-4b96-a59c-b132064a34c8","Type":"ContainerDied","Data":"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"}
Nov 03 01:40:44.085834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.085830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r" event={"ID":"d8f7cf05-97d2-4b96-a59c-b132064a34c8","Type":"ContainerDied","Data":"7e15db6c723687e962d9c2860f4f9ce9909663f38d4781616d47ad90cdf03b6e"}
Nov 03 01:40:44.085990 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.085857    4531 scope.go:122] "RemoveContainer" containerID="bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"
Nov 03 01:40:44.087471 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.087451    4531 generic.go:356] "Generic (PLEG): container finished" podID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerID="9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510" exitCode=0
Nov 03 01:40:44.087618 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.087602    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" event={"ID":"0d52c5b6-667d-4d5d-a11c-eccd90451b47","Type":"ContainerDied","Data":"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"}
Nov 03 01:40:44.087738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.087640    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth" event={"ID":"0d52c5b6-667d-4d5d-a11c-eccd90451b47","Type":"ContainerDied","Data":"9ed4c06e93f4a29237a33c5b26e22b28711ffe3ef0449fc290ec54f03fa04d7f"}
Nov 03 01:40:44.087738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.087721    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"
Nov 03 01:40:44.101540 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.101513    4531 scope.go:122] "RemoveContainer" containerID="bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"
Nov 03 01:40:44.101921 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:44.101886    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0\": container with ID starting with bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0 not found: ID does not exist" containerID="bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"
Nov 03 01:40:44.101921 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.101929    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0"} err="failed to get container status \"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0\": rpc error: code = NotFound desc = could not find container \"bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0\": container with ID starting with bb4d2ac7a6f48b49adc82b548fb0480de0ed36546c5e8042b5b1fae8c2d38fe0 not found: ID does not exist"
Nov 03 01:40:44.102096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.101955    4531 scope.go:122] "RemoveContainer" containerID="9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"
Nov 03 01:40:44.121233 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.121032    4531 scope.go:122] "RemoveContainer" containerID="9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"
Nov 03 01:40:44.123051 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:44.123025    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510\": container with ID starting with 9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510 not found: ID does not exist" containerID="9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"
Nov 03 01:40:44.123201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.123061    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510"} err="failed to get container status \"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510\": rpc error: code = NotFound desc = could not find container \"9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510\": container with ID starting with 9c06f9d11fce942b790ceea74ff0129aa992cf77aa7eac6fb57fdd37d054c510 not found: ID does not exist"
Nov 03 01:40:44.130650 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.130624    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:40:44.134183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.134146    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-l2qth"]
Nov 03 01:40:44.147313 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147288    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147329    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147473    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147498    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147536    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147632    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147655    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.147755 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.147676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\") pod \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\" (UID: \"d8f7cf05-97d2-4b96-a59c-b132064a34c8\") "
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148007    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kq4n\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-kube-api-access-8kq4n\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148022    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148032    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148041    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148049    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148057    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148066    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148074    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d52c5b6-667d-4d5d-a11c-eccd90451b47-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148082    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d52c5b6-667d-4d5d-a11c-eccd90451b47-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.148318 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148301    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.148920 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.148920 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148632    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:44.149040 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.148941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.149249 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.149222    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.151275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.151250    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "kube-api-access-fzvr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.151683 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.151643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:40:44.152368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.152346    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.153042 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.153022    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token" pod "d8f7cf05-97d2-4b96-a59c-b132064a34c8" (UID: "d8f7cf05-97d2-4b96-a59c-b132064a34c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.249829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249742    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.249829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249786    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.249829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249801    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.249829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249814    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.249829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249828    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.250155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249842    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f7cf05-97d2-4b96-a59c-b132064a34c8-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.250155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249855    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvr2\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-kube-api-access-fzvr2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.250155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249871    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.250155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.249884    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f7cf05-97d2-4b96-a59c-b132064a34c8-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.418842 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.418795    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:40:44.423731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.423694    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-gb65r"]
Nov 03 01:40:44.428291 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.428261    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:40:44.554780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.554691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.554780 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.554738    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.554828    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555000    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555044    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555122    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555220    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555329    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555353    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555440    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555498    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\") pod \"57f2ca1b-944c-4be3-9532-0a9c0120a625\" (UID: \"57f2ca1b-944c-4be3-9532-0a9c0120a625\") "
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.555565    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.556047    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.556069    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f2ca1b-944c-4be3-9532-0a9c0120a625-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.556083    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.556097    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.556515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.556138    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.558332 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.558304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.558810 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.558784    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:40:44.558904 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.558810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "kube-api-access-x7hgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:44.559330 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.559290    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo" pod "57f2ca1b-944c-4be3-9532-0a9c0120a625" (UID: "57f2ca1b-944c-4be3-9532-0a9c0120a625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:40:44.658289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.658238    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.658289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.658273    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.658289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.658283    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f2ca1b-944c-4be3-9532-0a9c0120a625-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.658289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.658292    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f2ca1b-944c-4be3-9532-0a9c0120a625-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.658289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.658302    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hgh\" (UniqueName: \"kubernetes.io/projected/57f2ca1b-944c-4be3-9532-0a9c0120a625-kube-api-access-x7hgh\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:44.905180 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.905135    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" path="/var/lib/kubelet/pods/0d52c5b6-667d-4d5d-a11c-eccd90451b47/volumes"
Nov 03 01:40:44.905721 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:44.905705    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" path="/var/lib/kubelet/pods/d8f7cf05-97d2-4b96-a59c-b132064a34c8/volumes"
Nov 03 01:40:45.143196 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:45.133100    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj" event={"ID":"57f2ca1b-944c-4be3-9532-0a9c0120a625","Type":"ContainerDied","Data":"5c8161301e6fd9bea87b9db41fc21636e06299cbe6245e3e8df0ba310e265c12"}
Nov 03 01:40:45.145047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:45.133389    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"
Nov 03 01:40:45.145047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:45.144941    4531 scope.go:122] "RemoveContainer" containerID="cab0cab3920a94d57233759b2bc7907d73362553312b49241ccf70a905b9ef99"
Nov 03 01:40:45.197419 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:45.196711    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:40:45.200302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:45.200171    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-531447-openshift-default-7995d46767-4xqgj"]
Nov 03 01:40:46.912603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:46.911294    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" path="/var/lib/kubelet/pods/57f2ca1b-944c-4be3-9532-0a9c0120a625/volumes"
Nov 03 01:40:50.992237 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:50.992209    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:40:50.992991 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:50.992957    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" podUID="341945e9-c2a6-4f1f-af71-647d40dc918e" containerName="maas-api" containerID="cri-o://8e40b11c359dd697c75e473191d62f85a17ad5bc641104fb9762bbfc9fbbcd54" gracePeriod=30
Nov 03 01:40:51.399835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.394926    4531 generic.go:356] "Generic (PLEG): container finished" podID="341945e9-c2a6-4f1f-af71-647d40dc918e" containerID="8e40b11c359dd697c75e473191d62f85a17ad5bc641104fb9762bbfc9fbbcd54" exitCode=0
Nov 03 01:40:51.399835 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.395189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" event={"ID":"341945e9-c2a6-4f1f-af71-647d40dc918e","Type":"ContainerDied","Data":"8e40b11c359dd697c75e473191d62f85a17ad5bc641104fb9762bbfc9fbbcd54"}
Nov 03 01:40:51.529331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.529305    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:40:51.599072 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599035    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\") pod \"341945e9-c2a6-4f1f-af71-647d40dc918e\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") "
Nov 03 01:40:51.599312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599130    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\") pod \"341945e9-c2a6-4f1f-af71-647d40dc918e\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") "
Nov 03 01:40:51.599312 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\" (UniqueName: \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\") pod \"341945e9-c2a6-4f1f-af71-647d40dc918e\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") "
Nov 03 01:40:51.599430 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599383    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\") pod \"341945e9-c2a6-4f1f-af71-647d40dc918e\" (UID: \"341945e9-c2a6-4f1f-af71-647d40dc918e\") "
Nov 03 01:40:51.599500 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599469    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca" pod "341945e9-c2a6-4f1f-af71-647d40dc918e" (UID: "341945e9-c2a6-4f1f-af71-647d40dc918e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:51.599963 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.599936    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/341945e9-c2a6-4f1f-af71-647d40dc918e-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:51.601906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.601877    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs" pod "341945e9-c2a6-4f1f-af71-647d40dc918e" (UID: "341945e9-c2a6-4f1f-af71-647d40dc918e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:51.602011 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.601978    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls" pod "341945e9-c2a6-4f1f-af71-647d40dc918e" (UID: "341945e9-c2a6-4f1f-af71-647d40dc918e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:51.603464 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.603434    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp" pod "341945e9-c2a6-4f1f-af71-647d40dc918e" (UID: "341945e9-c2a6-4f1f-af71-647d40dc918e"). InnerVolumeSpecName "kube-api-access-kbptp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:51.701453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.701399    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:51.701453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.701447    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/341945e9-c2a6-4f1f-af71-647d40dc918e-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:51.701667 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:51.701474    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbptp\" (UniqueName: \"kubernetes.io/projected/341945e9-c2a6-4f1f-af71-647d40dc918e-kube-api-access-kbptp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:52.403565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.403522    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"
Nov 03 01:40:52.404047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.403568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m" event={"ID":"341945e9-c2a6-4f1f-af71-647d40dc918e","Type":"ContainerDied","Data":"706c7f58c66f04fb67637409def52e7299410f41c323203c0fdba3a5a52d5667"}
Nov 03 01:40:52.404047 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.403641    4531 scope.go:122] "RemoveContainer" containerID="8e40b11c359dd697c75e473191d62f85a17ad5bc641104fb9762bbfc9fbbcd54"
Nov 03 01:40:52.427812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.427781    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:40:52.431266 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.431225    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-7c759a-675fc7dcdd-rwk8m"]
Nov 03 01:40:52.789078 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.788977    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:40:52.789315 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.789246    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" podUID="21649128-15d0-4cf6-b0a1-08bc8991951f" containerName="maas-api" containerID="cri-o://055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c" gracePeriod=30
Nov 03 01:40:52.907717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:52.907680    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341945e9-c2a6-4f1f-af71-647d40dc918e" path="/var/lib/kubelet/pods/341945e9-c2a6-4f1f-af71-647d40dc918e/volumes"
Nov 03 01:40:53.292276 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.292209    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:40:53.346458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.346427    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\") pod \"21649128-15d0-4cf6-b0a1-08bc8991951f\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") "
Nov 03 01:40:53.346685 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.346482    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\") pod \"21649128-15d0-4cf6-b0a1-08bc8991951f\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") "
Nov 03 01:40:53.346685 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.346519    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\" (UniqueName: \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\") pod \"21649128-15d0-4cf6-b0a1-08bc8991951f\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") "
Nov 03 01:40:53.346685 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.346542    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\") pod \"21649128-15d0-4cf6-b0a1-08bc8991951f\" (UID: \"21649128-15d0-4cf6-b0a1-08bc8991951f\") "
Nov 03 01:40:53.347296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.347197    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca" pod "21649128-15d0-4cf6-b0a1-08bc8991951f" (UID: "21649128-15d0-4cf6-b0a1-08bc8991951f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:53.347296 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.347276    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21649128-15d0-4cf6-b0a1-08bc8991951f-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:53.349173 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.349121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls" pod "21649128-15d0-4cf6-b0a1-08bc8991951f" (UID: "21649128-15d0-4cf6-b0a1-08bc8991951f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:53.349515 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.349494    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx" pod "21649128-15d0-4cf6-b0a1-08bc8991951f" (UID: "21649128-15d0-4cf6-b0a1-08bc8991951f"). InnerVolumeSpecName "kube-api-access-gxswx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:53.349620 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.349603    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs" pod "21649128-15d0-4cf6-b0a1-08bc8991951f" (UID: "21649128-15d0-4cf6-b0a1-08bc8991951f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:53.412055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.412023    4531 generic.go:356] "Generic (PLEG): container finished" podID="21649128-15d0-4cf6-b0a1-08bc8991951f" containerID="055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c" exitCode=0
Nov 03 01:40:53.412704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.412068    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" event={"ID":"21649128-15d0-4cf6-b0a1-08bc8991951f","Type":"ContainerDied","Data":"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"}
Nov 03 01:40:53.412704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.412088    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"
Nov 03 01:40:53.412704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.412186    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn" event={"ID":"21649128-15d0-4cf6-b0a1-08bc8991951f","Type":"ContainerDied","Data":"6433e43a0dd8dc8b044a3db0fb896fff09d05218c33426158bc870062f70e575"}
Nov 03 01:40:53.412704 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.412217    4531 scope.go:122] "RemoveContainer" containerID="055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"
Nov 03 01:40:53.428722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.428699    4531 scope.go:122] "RemoveContainer" containerID="055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"
Nov 03 01:40:53.429224 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:40:53.429134    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c\": container with ID starting with 055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c not found: ID does not exist" containerID="055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"
Nov 03 01:40:53.429224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.429185    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c"} err="failed to get container status \"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c\": rpc error: code = NotFound desc = could not find container \"055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c\": container with ID starting with 055ae5976e9fe2f67705bdceda9586d2b987f65a4f3cd203a1146fcdec137e0c not found: ID does not exist"
Nov 03 01:40:53.446615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.446581    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:40:53.448692 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.448662    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:53.448801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.448694    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21649128-15d0-4cf6-b0a1-08bc8991951f-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:53.448801 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.448711    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxswx\" (UniqueName: \"kubernetes.io/projected/21649128-15d0-4cf6-b0a1-08bc8991951f-kube-api-access-gxswx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:53.450098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:53.450066    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ef434a-77fc9f7986-6w9xn"]
Nov 03 01:40:54.904297 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:40:54.904260    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21649128-15d0-4cf6-b0a1-08bc8991951f" path="/var/lib/kubelet/pods/21649128-15d0-4cf6-b0a1-08bc8991951f/volumes"
Nov 03 01:41:05.071049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071013    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:05.071610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071594    4531 cpu_manager.go:414] "containerMap: removing container" podUID="341945e9-c2a6-4f1f-af71-647d40dc918e" containerName="maas-api"
Nov 03 01:41:05.071610 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071609    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341945e9-c2a6-4f1f-af71-647d40dc918e" containerName="maas-api"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071628    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071633    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071645    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d3e0b936-03da-4006-9d61-75118e093a9d" containerName="maas-api"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071650    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e0b936-03da-4006-9d61-75118e093a9d" containerName="maas-api"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071665    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071670    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071682    4531 cpu_manager.go:414] "containerMap: removing container" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071686    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071694    4531 cpu_manager.go:414] "containerMap: removing container" podUID="21649128-15d0-4cf6-b0a1-08bc8991951f" containerName="maas-api"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071698    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21649128-15d0-4cf6-b0a1-08bc8991951f" containerName="maas-api"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071704    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071709    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071720    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071724    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071731    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071735    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071743    4531 cpu_manager.go:414] "containerMap: removing container" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main"
Nov 03 01:41:05.071740 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071748    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071754    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071758    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071766    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071770    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071779    4531 cpu_manager.go:414] "containerMap: removing container" podUID="555ec9eb-2fab-419c-86c1-0db07e23907d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071783    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555ec9eb-2fab-419c-86c1-0db07e23907d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071794    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071799    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071804    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d03d2a92-b898-466b-a679-b94d45070b0d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071808    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03d2a92-b898-466b-a679-b94d45070b0d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071897    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c40d84-4cde-4104-84c5-cb356da785e4" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071907    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f2ca1b-944c-4be3-9532-0a9c0120a625" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071914    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="21649128-15d0-4cf6-b0a1-08bc8991951f" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071920    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b082ef0-f9b7-4096-a60c-0da0636c1846" containerName="payload-pre-processing"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071926    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ba8ade-6968-438c-b4f0-3ef913b858ad" containerName="main"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071933    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d52c5b6-667d-4d5d-a11c-eccd90451b47" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071938    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="675f003d-3ddc-4e7f-9743-00e7119b1ab5" containerName="main"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071943    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70ae258-f4f8-41a2-988b-74f9e7d75fc3" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071949    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a6a44e-0da1-4400-80cf-62dc78e274e1" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071954    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="555ec9eb-2fab-419c-86c1-0db07e23907d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071959    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e018a29c-f897-4e40-a2fa-94cb5b4c56a9" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071964    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03d2a92-b898-466b-a679-b94d45070b0d" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071970    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="341945e9-c2a6-4f1f-af71-647d40dc918e" containerName="maas-api"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071976    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f7cf05-97d2-4b96-a59c-b132064a34c8" containerName="istio-proxy"
Nov 03 01:41:05.072806 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.071982    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e0b936-03da-4006-9d61-75118e093a9d" containerName="maas-api"
Nov 03 01:41:05.075746 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.075712    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.108113 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.108077    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:05.141102 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.141067    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4a697f-openshift-default-dockercfg-wr6rf\""
Nov 03 01:41:05.245255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245208    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245381    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245556 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245617 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245700    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlgf\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245804    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245868    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.245962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.245906    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.346989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.346889    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.346989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.346961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347005    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlgf\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347067    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347095    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347149    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347633 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347325    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347649    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347687    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347819 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347751    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.347919 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.347880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.348121 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.348086    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.349791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.349765    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.349791 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.349778    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.355741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.355711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.356220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.356199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlgf\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\") pod \"e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") " pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.391451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.391417    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:05.685970 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.685933    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:05.686258 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:41:05.686234    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18770058_d434_4507_84e5_e7a64fdccb03.slice/crio-c5726256d23c6298e48f02e00ef294eeee657ca06d3cdce54f975d1518702018 WatchSource:0}: Error finding container c5726256d23c6298e48f02e00ef294eeee657ca06d3cdce54f975d1518702018: Status 404 returned error can't find the container with id c5726256d23c6298e48f02e00ef294eeee657ca06d3cdce54f975d1518702018
Nov 03 01:41:05.688744 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.688706    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:05.688848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.688790    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:05.688848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:05.688838    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:06.560400 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:06.560363    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" event={"ID":"18770058-d434-4507-84e5-e7a64fdccb03","Type":"ContainerStarted","Data":"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"}
Nov 03 01:41:06.560917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:06.560427    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" event={"ID":"18770058-d434-4507-84e5-e7a64fdccb03","Type":"ContainerStarted","Data":"c5726256d23c6298e48f02e00ef294eeee657ca06d3cdce54f975d1518702018"}
Nov 03 01:41:06.581476 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:06.581416    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podStartSLOduration=1.581399759 podStartE2EDuration="1.581399759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:06.579718572 +0000 UTC m=+3611.793462245" watchObservedRunningTime="2026-11-03 01:41:06.581399759 +0000 UTC m=+3611.795143427"
Nov 03 01:41:07.392537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:07.392503    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:07.394428 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:07.394396    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 01:41:07.394567 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:07.394460    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 01:41:08.072814 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.072775    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:41:08.078465 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.078427    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" podUID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" containerName="maas-api" containerID="cri-o://4ac5bdc1bb1f1293c7cb044add1e366fe3294606435eec5d9d954c5194a4d8e7" gracePeriod=30
Nov 03 01:41:08.405566 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.405463    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 01:41:08.405839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.405808    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 01:41:08.596207 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.595994    4531 generic.go:356] "Generic (PLEG): container finished" podID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" containerID="4ac5bdc1bb1f1293c7cb044add1e366fe3294606435eec5d9d954c5194a4d8e7" exitCode=0
Nov 03 01:41:08.596615 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.596527    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" event={"ID":"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa","Type":"ContainerDied","Data":"4ac5bdc1bb1f1293c7cb044add1e366fe3294606435eec5d9d954c5194a4d8e7"}
Nov 03 01:41:08.718590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.718559    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:41:08.814606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.814567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\") pod \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") "
Nov 03 01:41:08.814798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.814699    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\") pod \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") "
Nov 03 01:41:08.814798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.814745    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\") pod \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") "
Nov 03 01:41:08.814888 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.814797    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\" (UniqueName: \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\") pod \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\" (UID: \"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa\") "
Nov 03 01:41:08.815122 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.815097    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca" pod "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" (UID: "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:08.815320 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.815300    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:08.816738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.816705    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs" pod "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" (UID: "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:08.816846 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.816785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls" pod "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" (UID: "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:08.816940 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.816920    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8" pod "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" (UID: "b00ee078-1c60-43a7-bed4-61e7d7d1fdaa"). InnerVolumeSpecName "kube-api-access-gcwx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:08.916895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.916865    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:08.916895 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.916896    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwx8\" (UniqueName: \"kubernetes.io/projected/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-kube-api-access-gcwx8\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:08.917055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:08.916911    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:09.392897 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.392849    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 01:41:09.393460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.392939    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 01:41:09.607589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.607553    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr" event={"ID":"b00ee078-1c60-43a7-bed4-61e7d7d1fdaa","Type":"ContainerDied","Data":"46d2794c218d6088d14456caf425acc433db6a75cbba826b470a6016fe065f29"}
Nov 03 01:41:09.607862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.607763    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"
Nov 03 01:41:09.608021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.608006    4531 scope.go:122] "RemoveContainer" containerID="4ac5bdc1bb1f1293c7cb044add1e366fe3294606435eec5d9d954c5194a4d8e7"
Nov 03 01:41:09.671717 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.671602    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:41:09.674827 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:09.674792    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-57079b-946cf9869-thjcr"]
Nov 03 01:41:09.892858 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:09.892826    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:41:10.393989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:10.393950    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 01:41:10.394688 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:10.394653    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 01:41:10.905831 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:10.905798    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" path="/var/lib/kubelet/pods/b00ee078-1c60-43a7-bed4-61e7d7d1fdaa/volumes"
Nov 03 01:41:11.445190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:11.436129    4531 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Nov 03 01:41:11.445190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:11.436216    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Nov 03 01:41:12.419184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:12.410851    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:12.419184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:12.411350    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:12.419184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:12.412487    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:22.341788 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.341746    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 01:41:22.342381 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.342128    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" podUID="a86ecb79-989d-47cb-8661-998ad177164e" containerName="maas-api" containerID="cri-o://5254a3e9f4c8095582443e1331e06eab9bbb571e9cab126900f3ee45066ac9cb" gracePeriod=30
Nov 03 01:41:22.382153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.382112    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" podUID="a86ecb79-989d-47cb-8661-998ad177164e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.54:8443/health\": dial tcp 10.133.0.54:8443: connect: connection refused"
Nov 03 01:41:22.723236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.723115    4531 generic.go:356] "Generic (PLEG): container finished" podID="a86ecb79-989d-47cb-8661-998ad177164e" containerID="5254a3e9f4c8095582443e1331e06eab9bbb571e9cab126900f3ee45066ac9cb" exitCode=0
Nov 03 01:41:22.723236 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.723195    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" event={"ID":"a86ecb79-989d-47cb-8661-998ad177164e","Type":"ContainerDied","Data":"5254a3e9f4c8095582443e1331e06eab9bbb571e9cab126900f3ee45066ac9cb"}
Nov 03 01:41:22.852387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.852361    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 01:41:22.912977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.912943    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\") pod \"a86ecb79-989d-47cb-8661-998ad177164e\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") "
Nov 03 01:41:22.913183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.913119    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\") pod \"a86ecb79-989d-47cb-8661-998ad177164e\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") "
Nov 03 01:41:22.913282 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.913236    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\") pod \"a86ecb79-989d-47cb-8661-998ad177164e\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") "
Nov 03 01:41:22.913347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.913292    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\" (UniqueName: \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\") pod \"a86ecb79-989d-47cb-8661-998ad177164e\" (UID: \"a86ecb79-989d-47cb-8661-998ad177164e\") "
Nov 03 01:41:22.914532 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.914499    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca" pod "a86ecb79-989d-47cb-8661-998ad177164e" (UID: "a86ecb79-989d-47cb-8661-998ad177164e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:22.916004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.915981    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs" pod "a86ecb79-989d-47cb-8661-998ad177164e" (UID: "a86ecb79-989d-47cb-8661-998ad177164e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:22.916507 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.916482    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls" pod "a86ecb79-989d-47cb-8661-998ad177164e" (UID: "a86ecb79-989d-47cb-8661-998ad177164e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:22.916712 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:22.916695    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq" pod "a86ecb79-989d-47cb-8661-998ad177164e" (UID: "a86ecb79-989d-47cb-8661-998ad177164e"). InnerVolumeSpecName "kube-api-access-ljrwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:23.014932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.014843    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86ecb79-989d-47cb-8661-998ad177164e-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:23.014932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.014876    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:23.014932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.014886    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrwq\" (UniqueName: \"kubernetes.io/projected/a86ecb79-989d-47cb-8661-998ad177164e-kube-api-access-ljrwq\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:23.014932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.014896    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a86ecb79-989d-47cb-8661-998ad177164e-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:23.732798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.732766    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"
Nov 03 01:41:23.732798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.732794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w" event={"ID":"a86ecb79-989d-47cb-8661-998ad177164e","Type":"ContainerDied","Data":"b85dbb9d409ff2c4fcb03a97062202534444dda5e03c0bfc82c377e94eacaa89"}
Nov 03 01:41:23.733401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.732860    4531 scope.go:122] "RemoveContainer" containerID="5254a3e9f4c8095582443e1331e06eab9bbb571e9cab126900f3ee45066ac9cb"
Nov 03 01:41:23.759333 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.759306    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 01:41:23.764972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:23.764941    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f7680d-7f4588b9c8-qjb2w"]
Nov 03 01:41:24.905362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:24.905329    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86ecb79-989d-47cb-8661-998ad177164e" path="/var/lib/kubelet/pods/a86ecb79-989d-47cb-8661-998ad177164e/volumes"
Nov 03 01:41:34.002406 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.002372    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:41:34.002998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.002665    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" containerID="cri-o://57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec" gracePeriod=30
Nov 03 01:41:34.688574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.688532    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:34.689357 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689325    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" containerName="maas-api"
Nov 03 01:41:34.689521 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689361    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" containerName="maas-api"
Nov 03 01:41:34.689521 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689382    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a86ecb79-989d-47cb-8661-998ad177164e" containerName="maas-api"
Nov 03 01:41:34.689521 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689390    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86ecb79-989d-47cb-8661-998ad177164e" containerName="maas-api"
Nov 03 01:41:34.689722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689540    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00ee078-1c60-43a7-bed4-61e7d7d1fdaa" containerName="maas-api"
Nov 03 01:41:34.689722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.689553    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86ecb79-989d-47cb-8661-998ad177164e" containerName="maas-api"
Nov 03 01:41:34.693396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.693377    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.703227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.703205    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f7680d-openshift-default-dockercfg-fr542\""
Nov 03 01:41:34.707447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.707424    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:34.817132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817089    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817153    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817287    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssbp\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817351 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817336    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817489 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817399    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817489 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817577 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817577 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.817655 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.817607    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919242 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919203    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssbp\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919259    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919299    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919425    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919812 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.919931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.919886    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.920104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.920086    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.920227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.920135    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.920298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.920219    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.920492 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.920467    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.920578 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.920521    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.921678 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.921660    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.922075 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.922053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.936385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.936357    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:34.936716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:34.936663    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssbp\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\") pod \"e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") " pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:35.009928 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.009832    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:35.295544 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.295515    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:35.297859 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:41:35.297832    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714ac1f7_1b14_4755_b1dc_5278b5df6ced.slice/crio-8a82930173dbe8d68dd5844948f6adcaef069e5f43669826b955da7ba9f0168d WatchSource:0}: Error finding container 8a82930173dbe8d68dd5844948f6adcaef069e5f43669826b955da7ba9f0168d: Status 404 returned error can't find the container with id 8a82930173dbe8d68dd5844948f6adcaef069e5f43669826b955da7ba9f0168d
Nov 03 01:41:35.300427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.300391    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:35.300530 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.300459    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:35.300530 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.300504    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:41:35.829537 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.829495    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" event={"ID":"714ac1f7-1b14-4755-b1dc-5278b5df6ced","Type":"ContainerStarted","Data":"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"}
Nov 03 01:41:35.829729 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.829559    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" event={"ID":"714ac1f7-1b14-4755-b1dc-5278b5df6ced","Type":"ContainerStarted","Data":"8a82930173dbe8d68dd5844948f6adcaef069e5f43669826b955da7ba9f0168d"}
Nov 03 01:41:35.850879 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:35.850818    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podStartSLOduration=1.850804135 podStartE2EDuration="1.850804135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:35.848534105 +0000 UTC m=+3641.062277777" watchObservedRunningTime="2026-11-03 01:41:35.850804135 +0000 UTC m=+3641.064547807"
Nov 03 01:41:36.010289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:36.010237    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:36.011782 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:36.011757    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:36.011943 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:36.011805    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:37.011094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:37.011011    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:37.011094 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:37.011067    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:38.010486 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.010448    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:38.010668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.010510    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:38.122543 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.122505    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:41:38.122981 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.122765    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" podUID="3865d725-5e1d-4dd6-ab4c-071637eadfad" containerName="maas-api" containerID="cri-o://18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be" gracePeriod=30
Nov 03 01:41:38.768195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.766992    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:41:38.768195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.767056    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:41:38.776546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.775336    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:41:38.877665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.877569    4531 generic.go:356] "Generic (PLEG): container finished" podID="3865d725-5e1d-4dd6-ab4c-071637eadfad" containerID="18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be" exitCode=0
Nov 03 01:41:38.877665 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.877612    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"
Nov 03 01:41:38.877860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.877660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" event={"ID":"3865d725-5e1d-4dd6-ab4c-071637eadfad","Type":"ContainerDied","Data":"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"}
Nov 03 01:41:38.877860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.877723    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27" event={"ID":"3865d725-5e1d-4dd6-ab4c-071637eadfad","Type":"ContainerDied","Data":"4e6bf70a165b3d9e84d47fe794564e761ab2a925ce99ecc1dc6b56b10affa2af"}
Nov 03 01:41:38.877860 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.877755    4531 scope.go:122] "RemoveContainer" containerID="18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"
Nov 03 01:41:38.887938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.887909    4531 scope.go:122] "RemoveContainer" containerID="18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"
Nov 03 01:41:38.888259 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:38.888237    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be\": container with ID starting with 18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be not found: ID does not exist" containerID="18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"
Nov 03 01:41:38.888329 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.888270    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be"} err="failed to get container status \"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be\": rpc error: code = NotFound desc = could not find container \"18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be\": container with ID starting with 18557be6b8c79b0709874a4835618fd3e01f89de9f7daf0b3859cd3bbef857be not found: ID does not exist"
Nov 03 01:41:38.922427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.922393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\") pod \"3865d725-5e1d-4dd6-ab4c-071637eadfad\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") "
Nov 03 01:41:38.922603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.922544    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\") pod \"3865d725-5e1d-4dd6-ab4c-071637eadfad\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") "
Nov 03 01:41:38.922603 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.922586    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\" (UniqueName: \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\") pod \"3865d725-5e1d-4dd6-ab4c-071637eadfad\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") "
Nov 03 01:41:38.922701 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.922674    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") pod \"3865d725-5e1d-4dd6-ab4c-071637eadfad\" (UID: \"3865d725-5e1d-4dd6-ab4c-071637eadfad\") "
Nov 03 01:41:38.922805 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.922785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca" pod "3865d725-5e1d-4dd6-ab4c-071637eadfad" (UID: "3865d725-5e1d-4dd6-ab4c-071637eadfad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:38.923327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.923302    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3865d725-5e1d-4dd6-ab4c-071637eadfad-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:38.924750 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.924723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs" pod "3865d725-5e1d-4dd6-ab4c-071637eadfad" (UID: "3865d725-5e1d-4dd6-ab4c-071637eadfad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:38.924848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.924766    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls" pod "3865d725-5e1d-4dd6-ab4c-071637eadfad" (UID: "3865d725-5e1d-4dd6-ab4c-071637eadfad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:38.924848 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:38.924814    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2" pod "3865d725-5e1d-4dd6-ab4c-071637eadfad" (UID: "3865d725-5e1d-4dd6-ab4c-071637eadfad"). InnerVolumeSpecName "kube-api-access-c5zl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:39.010344 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.010303    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:39.010570 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.010378    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:39.024722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.024621    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.024722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.024655    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zl2\" (UniqueName: \"kubernetes.io/projected/3865d725-5e1d-4dd6-ab4c-071637eadfad-kube-api-access-c5zl2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.024722 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.024674    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3865d725-5e1d-4dd6-ab4c-071637eadfad-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.306671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.306614    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:41:39.310046 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.310013    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4a697f-6b89ff6486-4wg27"]
Nov 03 01:41:39.507186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.507142    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:41:39.685183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685026    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685176    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685212    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685273    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685524 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685426    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685544    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.685762 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.685573    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\") pod \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\" (UID: \"ba6f631a-2510-4c8e-9f9d-791087aa43e1\") "
Nov 03 01:41:39.699227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.686879    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:39.699227 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.687198    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:39.701612 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.701568    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:39.704234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.703012    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:39.704234 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.703443    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:39.707298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.706217    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "kube-api-access-hrl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:39.707298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.706270    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:39.707298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.706556    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:41:39.707298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.707242    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy" pod "ba6f631a-2510-4c8e-9f9d-791087aa43e1" (UID: "ba6f631a-2510-4c8e-9f9d-791087aa43e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795451    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrl6m\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-kube-api-access-hrl6m\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795494    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795523    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795536    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795550    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795562    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795575    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f631a-2510-4c8e-9f9d-791087aa43e1-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795588    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.797014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.795603    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f631a-2510-4c8e-9f9d-791087aa43e1-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:39.918184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.917110    4531 generic.go:356] "Generic (PLEG): container finished" podID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerID="57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec" exitCode=0
Nov 03 01:41:39.918184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.917196    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" event={"ID":"ba6f631a-2510-4c8e-9f9d-791087aa43e1","Type":"ContainerDied","Data":"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"}
Nov 03 01:41:39.918184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.917259    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd" event={"ID":"ba6f631a-2510-4c8e-9f9d-791087aa43e1","Type":"ContainerDied","Data":"8a96e9d841e7d15d8ea15feb1497dd234ac12627bed5530bfd6bfe377a82b875"}
Nov 03 01:41:39.918184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.917290    4531 scope.go:122] "RemoveContainer" containerID="57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"
Nov 03 01:41:39.918184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.918010    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"
Nov 03 01:41:39.965186 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:39.962904    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:41:39.966510 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.966420    4531 scope.go:122] "RemoveContainer" containerID="57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"
Nov 03 01:41:39.970181 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:39.966824    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec\": container with ID starting with 57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec not found: ID does not exist" containerID="57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"
Nov 03 01:41:39.970181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.966865    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec"} err="failed to get container status \"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec\": rpc error: code = NotFound desc = could not find container \"57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec\": container with ID starting with 57f1803f3233718e1a059b4a9c91c4986b19fa7172df344363fc0ff3434d27ec not found: ID does not exist"
Nov 03 01:41:39.970181 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.968879    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:41:39.980290 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:39.977285    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-717217-openshift-default-578c69554c-n9hbd"]
Nov 03 01:41:40.012186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:40.011232    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:40.012186 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:40.011293    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:40.903586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:40.903552    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3865d725-5e1d-4dd6-ab4c-071637eadfad" path="/var/lib/kubelet/pods/3865d725-5e1d-4dd6-ab4c-071637eadfad/volumes"
Nov 03 01:41:40.904054 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:40.904040    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" path="/var/lib/kubelet/pods/ba6f631a-2510-4c8e-9f9d-791087aa43e1/volumes"
Nov 03 01:41:41.010408 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:41.010374    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:41.010645 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:41.010431    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:42.010997 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:42.010960    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:41:42.011544 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:42.011020    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:41:43.458198 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:43.458151    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:43.459508 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:43.459477    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:43.460497 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:43.460474    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:44.196766 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:44.196729    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:44.197014 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:44.196981    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy" containerID="cri-o://1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f" gracePeriod=30
Nov 03 01:41:45.548970 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:45.548284    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:41:45.548970 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:45.548576    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" podUID="00535132-f427-428e-9b3d-301eeace95db" containerName="maas-api" containerID="cri-o://2dfa38a7dc220be4bcba5bd8c02343acdde8327e041bba04861d83b775560eea" gracePeriod=30
Nov 03 01:41:46.025265 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.025233    4531 generic.go:356] "Generic (PLEG): container finished" podID="00535132-f427-428e-9b3d-301eeace95db" containerID="2dfa38a7dc220be4bcba5bd8c02343acdde8327e041bba04861d83b775560eea" exitCode=0
Nov 03 01:41:46.025427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.025306    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" event={"ID":"00535132-f427-428e-9b3d-301eeace95db","Type":"ContainerDied","Data":"2dfa38a7dc220be4bcba5bd8c02343acdde8327e041bba04861d83b775560eea"}
Nov 03 01:41:46.025427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.025401    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr" event={"ID":"00535132-f427-428e-9b3d-301eeace95db","Type":"ContainerDied","Data":"9c683f7be6fcdc498f1e99b4d814a816896a4ab8f9c5755e32bff7f943bb8251"}
Nov 03 01:41:46.025427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.025423    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c683f7be6fcdc498f1e99b4d814a816896a4ab8f9c5755e32bff7f943bb8251"
Nov 03 01:41:46.029728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.029700    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:41:46.038369 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.038344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\" (UniqueName: \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\") pod \"00535132-f427-428e-9b3d-301eeace95db\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") "
Nov 03 01:41:46.038512 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.038403    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") pod \"00535132-f427-428e-9b3d-301eeace95db\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") "
Nov 03 01:41:46.038663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.038644    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\") pod \"00535132-f427-428e-9b3d-301eeace95db\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") "
Nov 03 01:41:46.038736 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.038689    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\") pod \"00535132-f427-428e-9b3d-301eeace95db\" (UID: \"00535132-f427-428e-9b3d-301eeace95db\") "
Nov 03 01:41:46.039013 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.038974    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca" pod "00535132-f427-428e-9b3d-301eeace95db" (UID: "00535132-f427-428e-9b3d-301eeace95db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:46.039258 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.039232    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00535132-f427-428e-9b3d-301eeace95db-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:46.040868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.040841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx" pod "00535132-f427-428e-9b3d-301eeace95db" (UID: "00535132-f427-428e-9b3d-301eeace95db"). InnerVolumeSpecName "kube-api-access-vlprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:46.041104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.041082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls" pod "00535132-f427-428e-9b3d-301eeace95db" (UID: "00535132-f427-428e-9b3d-301eeace95db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:46.041250 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.041226    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs" pod "00535132-f427-428e-9b3d-301eeace95db" (UID: "00535132-f427-428e-9b3d-301eeace95db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:46.140741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.140703    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:46.140741 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.140742    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlprx\" (UniqueName: \"kubernetes.io/projected/00535132-f427-428e-9b3d-301eeace95db-kube-api-access-vlprx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:46.140964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.140759    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00535132-f427-428e-9b3d-301eeace95db-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:46.317962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.317873    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:46.318143 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:46.318117    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy" containerID="cri-o://a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476" gracePeriod=30
Nov 03 01:41:47.033085 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:47.033043    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"
Nov 03 01:41:47.056301 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:47.056270    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:41:47.060611 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:47.060584    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-556c14-695bffddf4-ssfkr"]
Nov 03 01:41:48.903354 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:48.903321    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00535132-f427-428e-9b3d-301eeace95db" path="/var/lib/kubelet/pods/00535132-f427-428e-9b3d-301eeace95db/volumes"
Nov 03 01:41:49.507972 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.501349    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:49.594459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594420    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594472    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594509    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594536    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594566    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594661 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594592    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.594931 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.594908    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:49.595285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595235    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:49.595432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.595432 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595346    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.595551 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595455    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\") pod \"18770058-d434-4507-84e5-e7a64fdccb03\" (UID: \"18770058-d434-4507-84e5-e7a64fdccb03\") "
Nov 03 01:41:49.595843 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595783    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:49.596238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.595972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:49.596238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.596117    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.596238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.596139    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.596238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.596154    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.596238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.596215    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.596565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.596454    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:49.597440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.597397    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:49.597980 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.597930    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:41:49.598096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.597955    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "kube-api-access-gvlgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:49.598774 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.598748    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token" pod "18770058-d434-4507-84e5-e7a64fdccb03" (UID: "18770058-d434-4507-84e5-e7a64fdccb03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:49.695210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.695154    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 01:41:49.698136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.698033    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18770058-d434-4507-84e5-e7a64fdccb03-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.698136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.698066    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18770058-d434-4507-84e5-e7a64fdccb03-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.698136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.698081    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18770058-d434-4507-84e5-e7a64fdccb03-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.698136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.698097    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvlgf\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-kube-api-access-gvlgf\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.698136 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.698114    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18770058-d434-4507-84e5-e7a64fdccb03-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:49.701832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:49.701794    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 01:41:50.057116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.056564    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"
Nov 03 01:41:50.057116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.056950    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" event={"ID":"18770058-d434-4507-84e5-e7a64fdccb03","Type":"ContainerDied","Data":"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"}
Nov 03 01:41:50.057116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.057037    4531 scope.go:122] "RemoveContainer" containerID="1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"
Nov 03 01:41:50.057795 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.057736    4531 generic.go:356] "Generic (PLEG): container finished" podID="18770058-d434-4507-84e5-e7a64fdccb03" containerID="1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f" exitCode=0
Nov 03 01:41:50.057856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.057834    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz" event={"ID":"18770058-d434-4507-84e5-e7a64fdccb03","Type":"ContainerDied","Data":"c5726256d23c6298e48f02e00ef294eeee657ca06d3cdce54f975d1518702018"}
Nov 03 01:41:50.068840 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.068818    4531 scope.go:122] "RemoveContainer" containerID="1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"
Nov 03 01:41:50.069262 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:50.069233    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f\": container with ID starting with 1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f not found: ID does not exist" containerID="1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"
Nov 03 01:41:50.069324 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.069278    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f"} err="failed to get container status \"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f\": rpc error: code = NotFound desc = could not find container \"1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f\": container with ID starting with 1bb484ee0bcf7b3410bb01624414a445ee29991dc1a250774052ff176965671f not found: ID does not exist"
Nov 03 01:41:50.083484 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.083454    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:50.086907 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.086882    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4a697f-openshift-default-666db9f8f5-79hdz"]
Nov 03 01:41:50.904423 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:50.904381    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18770058-d434-4507-84e5-e7a64fdccb03" path="/var/lib/kubelet/pods/18770058-d434-4507-84e5-e7a64fdccb03/volumes"
Nov 03 01:41:51.619346 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.619283    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:51.726969 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.726921    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.726969 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.726968    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727012    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727074    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727141    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727241 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727202    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727393 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727248    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727393 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727517 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727410    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\") pod \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\" (UID: \"714ac1f7-1b14-4755-b1dc-5278b5df6ced\") "
Nov 03 01:41:51.727643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727612    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:51.727643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727632    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:51.727841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727817    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:51.727960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727837    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:51.727960 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.727863    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:51.728093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.728019    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.728093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.728039    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.728093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.728055    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.728093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.728070    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.728093 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.728083    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.729336 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.729313    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:41:51.729649 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.729627    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:41:51.729727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.729651    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "kube-api-access-vssbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:51.732554 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.732527    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token" pod "714ac1f7-1b14-4755-b1dc-5278b5df6ced" (UID: "714ac1f7-1b14-4755-b1dc-5278b5df6ced"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:51.829208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.829177    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.829208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.829206    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.829208 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.829216    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssbp\" (UniqueName: \"kubernetes.io/projected/714ac1f7-1b14-4755-b1dc-5278b5df6ced-kube-api-access-vssbp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:51.829447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:51.829226    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714ac1f7-1b14-4755-b1dc-5278b5df6ced-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:52.075957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.075913    4531 generic.go:356] "Generic (PLEG): container finished" podID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerID="a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476" exitCode=0
Nov 03 01:41:52.076193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.075977    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"
Nov 03 01:41:52.076193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.075995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" event={"ID":"714ac1f7-1b14-4755-b1dc-5278b5df6ced","Type":"ContainerDied","Data":"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"}
Nov 03 01:41:52.076193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.076041    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr" event={"ID":"714ac1f7-1b14-4755-b1dc-5278b5df6ced","Type":"ContainerDied","Data":"8a82930173dbe8d68dd5844948f6adcaef069e5f43669826b955da7ba9f0168d"}
Nov 03 01:41:52.076193 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.076062    4531 scope.go:122] "RemoveContainer" containerID="a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"
Nov 03 01:41:52.086640 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.086622    4531 scope.go:122] "RemoveContainer" containerID="a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"
Nov 03 01:41:52.086912 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:52.086892    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476\": container with ID starting with a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476 not found: ID does not exist" containerID="a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"
Nov 03 01:41:52.086969 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.086920    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476"} err="failed to get container status \"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476\": rpc error: code = NotFound desc = could not find container \"a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476\": container with ID starting with a647e3937a09a32b4bac84968cfc01319b692c860cd09a36156f2ff255909476 not found: ID does not exist"
Nov 03 01:41:52.099104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.099080    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:52.104654 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.104630    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f7680d-openshift-default-c4488d5c9-d7cxr"]
Nov 03 01:41:52.902886 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:52.902853    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" path="/var/lib/kubelet/pods/714ac1f7-1b14-4755-b1dc-5278b5df6ced/volumes"
Nov 03 01:41:56.170702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.170665    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:41:56.171434 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.170898    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" podUID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" containerName="maas-api" containerID="cri-o://5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98" gracePeriod=30
Nov 03 01:41:56.583609 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.583577    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:41:56.705643 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.705609    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\") pod \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") "
Nov 03 01:41:56.705851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.705694    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\" (UniqueName: \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\") pod \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") "
Nov 03 01:41:56.705851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.705729    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\") pod \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") "
Nov 03 01:41:56.705851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.705789    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\") pod \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\" (UID: \"9a1aeaf6-086e-49dc-900c-18870ba9ae0c\") "
Nov 03 01:41:56.706284 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.706257    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca" pod "9a1aeaf6-086e-49dc-900c-18870ba9ae0c" (UID: "9a1aeaf6-086e-49dc-900c-18870ba9ae0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:56.707833 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.707807    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs" pod "9a1aeaf6-086e-49dc-900c-18870ba9ae0c" (UID: "9a1aeaf6-086e-49dc-900c-18870ba9ae0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:56.707961 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.707948    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls" pod "9a1aeaf6-086e-49dc-900c-18870ba9ae0c" (UID: "9a1aeaf6-086e-49dc-900c-18870ba9ae0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:56.708027 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.707991    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx" pod "9a1aeaf6-086e-49dc-900c-18870ba9ae0c" (UID: "9a1aeaf6-086e-49dc-900c-18870ba9ae0c"). InnerVolumeSpecName "kube-api-access-9s9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:56.807259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.807176    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:56.807259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.807209    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:56.807259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.807220    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9lx\" (UniqueName: \"kubernetes.io/projected/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-kube-api-access-9s9lx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:56.807259 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:56.807229    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a1aeaf6-086e-49dc-900c-18870ba9ae0c-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:57.129977 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.129940    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" containerID="5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98" exitCode=0
Nov 03 01:41:57.130235 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.129990    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"
Nov 03 01:41:57.130235 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.129988    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" event={"ID":"9a1aeaf6-086e-49dc-900c-18870ba9ae0c","Type":"ContainerDied","Data":"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"}
Nov 03 01:41:57.130235 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.130133    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz" event={"ID":"9a1aeaf6-086e-49dc-900c-18870ba9ae0c","Type":"ContainerDied","Data":"079096b84eb9ae9054af1f389427cd766cadfaece6df1e735e36962246e7c3e2"}
Nov 03 01:41:57.130235 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.130201    4531 scope.go:122] "RemoveContainer" containerID="5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"
Nov 03 01:41:57.142770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.142746    4531 scope.go:122] "RemoveContainer" containerID="5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"
Nov 03 01:41:57.143110 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:41:57.143087    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98\": container with ID starting with 5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98 not found: ID does not exist" containerID="5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"
Nov 03 01:41:57.143215 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.143118    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98"} err="failed to get container status \"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98\": rpc error: code = NotFound desc = could not find container \"5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98\": container with ID starting with 5dfa32add0dbe6aeac211471b4f06ebfd90c9d17667da7c244461dcff1e33e98 not found: ID does not exist"
Nov 03 01:41:57.149067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.149042    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:41:57.153642 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.153616    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-821f5a-7c4959889f-vz5rz"]
Nov 03 01:41:57.558882 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:57.558804    4531 scope.go:122] "RemoveContainer" containerID="2dfa38a7dc220be4bcba5bd8c02343acdde8327e041bba04861d83b775560eea"
Nov 03 01:41:58.902785 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:58.902748    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" path="/var/lib/kubelet/pods/9a1aeaf6-086e-49dc-900c-18870ba9ae0c/volumes"
Nov 03 01:41:59.855380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:59.855336    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:41:59.855605 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:41:59.855579    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" containerID="cri-o://498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094" gracePeriod=30
Nov 03 01:42:01.298081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:01.298041    4531 patch_prober.go:29] interesting pod/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:42:01.298574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:01.298108    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:42:03.802909 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:03.802867    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:42:03.803460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:03.803131    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" containerID="cri-o://7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127" gracePeriod=30
Nov 03 01:42:03.806396 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:03.806363    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:42:03.806621 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:03.806591    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy" containerID="cri-o://a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc" gracePeriod=30
Nov 03 01:42:04.116058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:04.116016    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-lsz2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:42:04.116274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:04.116084    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:42:04.725594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:04.725542    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main" containerID="cri-o://ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0" gracePeriod=45
Nov 03 01:42:04.731829 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:04.731793    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main" containerID="cri-o://762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d" gracePeriod=45
Nov 03 01:42:05.039401 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.039122    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 01:42:05.090792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.089499    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:42:05.170962 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.170923    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.171231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171004    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.171231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171058    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.171231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171097    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.171231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171151    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\" (UniqueName: \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171237    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171267    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171338    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171403    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\") pod \"f037a05a-daaf-4185-9405-2fd5e4112f6a\" (UID: \"f037a05a-daaf-4185-9405-2fd5e4112f6a\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171447    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\" (UniqueName: \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171488    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\") pod \"0e524439-cc34-4594-8570-64ba80a8ea8d\" (UID: \"0e524439-cc34-4594-8570-64ba80a8ea8d\") "
Nov 03 01:42:05.172029 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.171916    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.172522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172054    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.172522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172129    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.172522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172285    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.172522 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.173035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172903    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.173035 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.172965    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.175273 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.175234    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "kube-api-access-dstqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:05.175685 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.175625    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.176201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.176153    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "kube-api-access-5jq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:05.176354 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.176332    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs" pod "f037a05a-daaf-4185-9405-2fd5e4112f6a" (UID: "f037a05a-daaf-4185-9405-2fd5e4112f6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:05.176354 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.176341    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.177750 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.177724    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs" pod "0e524439-cc34-4594-8570-64ba80a8ea8d" (UID: "0e524439-cc34-4594-8570-64ba80a8ea8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:05.189792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.189767    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:42:05.198457 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.198432    4531 generic.go:356] "Generic (PLEG): container finished" podID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerID="ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0" exitCode=0
Nov 03 01:42:05.198597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.198528    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" event={"ID":"f037a05a-daaf-4185-9405-2fd5e4112f6a","Type":"ContainerDied","Data":"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"}
Nov 03 01:42:05.198597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.198566    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn" event={"ID":"f037a05a-daaf-4185-9405-2fd5e4112f6a","Type":"ContainerDied","Data":"23dad678fef0f6ff4e1ce4dee9eadb654d47e8d1a3e2c45a211c3b24826b281f"}
Nov 03 01:42:05.198597 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.198588    4531 scope.go:122] "RemoveContainer" containerID="ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"
Nov 03 01:42:05.198773 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.198721    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"
Nov 03 01:42:05.201008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.200983    4531 generic.go:356] "Generic (PLEG): container finished" podID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerID="762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d" exitCode=0
Nov 03 01:42:05.201117 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.201028    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"
Nov 03 01:42:05.201209 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.201156    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" event={"ID":"0e524439-cc34-4594-8570-64ba80a8ea8d","Type":"ContainerDied","Data":"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"}
Nov 03 01:42:05.201281 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.201228    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq" event={"ID":"0e524439-cc34-4594-8570-64ba80a8ea8d","Type":"ContainerDied","Data":"da4a83f29600a6349fdd91980203784a46d9fa8ac4e2bd3666aa45b358fa1a81"}
Nov 03 01:42:05.204245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.202988    4531 generic.go:356] "Generic (PLEG): container finished" podID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerID="498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094" exitCode=0
Nov 03 01:42:05.204245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.203033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" event={"ID":"f21dd413-d51f-4eaf-9f47-12eab1686a34","Type":"ContainerDied","Data":"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"}
Nov 03 01:42:05.204245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.203074    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz" event={"ID":"f21dd413-d51f-4eaf-9f47-12eab1686a34","Type":"ContainerDied","Data":"30bad4ba448ed039913d63ec1467120bba22e6d08f8d3d033b8aff7afab58359"}
Nov 03 01:42:05.204245 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.203135    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"
Nov 03 01:42:05.224830 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.222267    4531 scope.go:122] "RemoveContainer" containerID="ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"
Nov 03 01:42:05.261814 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:05.261718    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0\": container with ID starting with ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0 not found: ID does not exist" containerID="ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"
Nov 03 01:42:05.261814 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.261765    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0"} err="failed to get container status \"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0\": rpc error: code = NotFound desc = could not find container \"ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0\": container with ID starting with ba8a1cdccd39928174e8c51fcdd08cd8af4c485ccbbdbce08a3f49c448de4ef0 not found: ID does not exist"
Nov 03 01:42:05.261814 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.261789    4531 scope.go:122] "RemoveContainer" containerID="762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.272757    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.273260    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.273778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.274262    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.274679    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.274744    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.274825    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.275608    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.275693    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.275729    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.275824    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.275890    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\") pod \"f21dd413-d51f-4eaf-9f47-12eab1686a34\" (UID: \"f21dd413-d51f-4eaf-9f47-12eab1686a34\") "
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276278    4531 scope.go:122] "RemoveContainer" containerID="762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276432    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstqt\" (UniqueName: \"kubernetes.io/projected/0e524439-cc34-4594-8570-64ba80a8ea8d-kube-api-access-dstqt\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276456    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e524439-cc34-4594-8570-64ba80a8ea8d-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276470    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276485    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276498    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f037a05a-daaf-4185-9405-2fd5e4112f6a-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276511    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276524    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jq4d\" (UniqueName: \"kubernetes.io/projected/f037a05a-daaf-4185-9405-2fd5e4112f6a-kube-api-access-5jq4d\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276538    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f21dd413-d51f-4eaf-9f47-12eab1686a34-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276552    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276566    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276578    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e524439-cc34-4594-8570-64ba80a8ea8d-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276592    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276605    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276618    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f037a05a-daaf-4185-9405-2fd5e4112f6a-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:05.276620    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d\": container with ID starting with 762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d not found: ID does not exist" containerID="762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276656    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d"} err="failed to get container status \"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d\": rpc error: code = NotFound desc = could not find container \"762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d\": container with ID starting with 762737d0347c33badb27b7097a045063f09bb0b46a2eb77e54c8ffaa06ee587d not found: ID does not exist"
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.276680    4531 scope.go:122] "RemoveContainer" containerID="498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.277187    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.279453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.277436    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.281892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.279938    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:05.284238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.284210    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:05.284438 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.284415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:05.285347 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.285325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj" pod "f21dd413-d51f-4eaf-9f47-12eab1686a34" (UID: "f21dd413-d51f-4eaf-9f47-12eab1686a34"). InnerVolumeSpecName "kube-api-access-thscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:05.292362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.292326    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 01:42:05.297957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.296297    4531 scope.go:122] "RemoveContainer" containerID="498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"
Nov 03 01:42:05.297957 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:05.296668    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094\": container with ID starting with 498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094 not found: ID does not exist" containerID="498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"
Nov 03 01:42:05.297957 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.296709    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094"} err="failed to get container status \"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094\": rpc error: code = NotFound desc = could not find container \"498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094\": container with ID starting with 498a19d75b8af557c4a2c9a3bec359969848044c5adc54f44c82970c20e7b094 not found: ID does not exist"
Nov 03 01:42:05.301851 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.299100    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-w4-650be0-kserve-5d8bfcc5dc-48xjq"]
Nov 03 01:42:05.306815 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.306793    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 01:42:05.311298 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.311272    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-650be0/distinct-2-w4-650be0-kserve-8456c84f5b-68dkn"]
Nov 03 01:42:05.377713 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377674    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.377869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377731    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thscj\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-kube-api-access-thscj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.377869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377744    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.377869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377753    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f21dd413-d51f-4eaf-9f47-12eab1686a34-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.377869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377763    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.377869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.377771    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f21dd413-d51f-4eaf-9f47-12eab1686a34-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:05.527246 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.527147    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:42:05.530775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:05.530745    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-821f5a-openshift-default-7bfc657d9-z6zqz"]
Nov 03 01:42:06.905060 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:06.905018    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" path="/var/lib/kubelet/pods/0e524439-cc34-4594-8570-64ba80a8ea8d/volumes"
Nov 03 01:42:06.905693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:06.905672    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" path="/var/lib/kubelet/pods/f037a05a-daaf-4185-9405-2fd5e4112f6a/volumes"
Nov 03 01:42:06.906323 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:06.906299    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" path="/var/lib/kubelet/pods/f21dd413-d51f-4eaf-9f47-12eab1686a34/volumes"
Nov 03 01:42:07.769770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:07.769701    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:42:07.770254 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:07.770219    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" podUID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" containerName="maas-api" containerID="cri-o://d49823d77d876a1431f8c73262e1c689c8f6161b4cbbf7e1f40b54a626562467" gracePeriod=30
Nov 03 01:42:08.228475 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.228442    4531 generic.go:356] "Generic (PLEG): container finished" podID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" containerID="d49823d77d876a1431f8c73262e1c689c8f6161b4cbbf7e1f40b54a626562467" exitCode=0
Nov 03 01:42:08.229067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.228496    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" event={"ID":"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778","Type":"ContainerDied","Data":"d49823d77d876a1431f8c73262e1c689c8f6161b4cbbf7e1f40b54a626562467"}
Nov 03 01:42:08.294641 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.294615    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:42:08.364052 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.364013    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\") pod \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") "
Nov 03 01:42:08.364261 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.364089    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\" (UniqueName: \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\") pod \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") "
Nov 03 01:42:08.364362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.364269    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\") pod \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") "
Nov 03 01:42:08.364362 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.364317    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") pod \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\" (UID: \"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778\") "
Nov 03 01:42:08.365426 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.365394    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca" pod "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" (UID: "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:08.367998 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.367974    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv" pod "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" (UID: "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778"). InnerVolumeSpecName "kube-api-access-z28lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:08.370607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.370580    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls" pod "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" (UID: "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:08.371775 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.371078    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs" pod "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" (UID: "5922c0ec-e00e-4d6a-b0a6-08ff53bbb778"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:08.465862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.465763    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:08.465862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.465797    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z28lv\" (UniqueName: \"kubernetes.io/projected/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-kube-api-access-z28lv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:08.465862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.465808    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:08.465862 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:08.465817    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.109834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.109499    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:42:09.151996 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.151964    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:42:09.178067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178090    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178145    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178264 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178219    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178329    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178371    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178413    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178462 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178449    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178483    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178510    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178556    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178585    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178616    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178664    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.178738 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178696    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.179151 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178747    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\") pod \"27f34cf8-614c-4856-8711-5f980c6a6ebe\" (UID: \"27f34cf8-614c-4856-8711-5f980c6a6ebe\") "
Nov 03 01:42:09.179151 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.178799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\") pod \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\" (UID: \"74d4dcef-ba0d-4f3b-8c18-8557274b30e0\") "
Nov 03 01:42:09.179528 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.179497    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.179739 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.179713    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.180126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.179850    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.180126 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.179935    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:09.180449 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.180217    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.180565 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.180456    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.180778 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.180753    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.183496 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.183473    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.183934 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.183902    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:09.184574 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.184546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:09.184682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.184662    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:09.184749 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.184671    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.184799 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.184769    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "kube-api-access-v5ms8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:09.184855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.184828    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:09.186275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.186245    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:09.186444 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.186425    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "kube-api-access-bwjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:09.187310 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.187252    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy" pod "74d4dcef-ba0d-4f3b-8c18-8557274b30e0" (UID: "74d4dcef-ba0d-4f3b-8c18-8557274b30e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.187899 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.187878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy" pod "27f34cf8-614c-4856-8711-5f980c6a6ebe" (UID: "27f34cf8-614c-4856-8711-5f980c6a6ebe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:09.236808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.236769    4531 generic.go:356] "Generic (PLEG): container finished" podID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerID="a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc" exitCode=0
Nov 03 01:42:09.237304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.236826    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"
Nov 03 01:42:09.237304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.236840    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" event={"ID":"74d4dcef-ba0d-4f3b-8c18-8557274b30e0","Type":"ContainerDied","Data":"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"}
Nov 03 01:42:09.237304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.236893    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c" event={"ID":"74d4dcef-ba0d-4f3b-8c18-8557274b30e0","Type":"ContainerDied","Data":"d9123e7d40814eaa99a6569845ec968603ae884fea3e5b290523b09e2f0f4dce"}
Nov 03 01:42:09.237304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.236916    4531 scope.go:122] "RemoveContainer" containerID="a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.238633    4531 generic.go:356] "Generic (PLEG): container finished" podID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerID="7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127" exitCode=0
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.238663    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" event={"ID":"27f34cf8-614c-4856-8711-5f980c6a6ebe","Type":"ContainerDied","Data":"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"}
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.238692    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.238723    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g" event={"ID":"27f34cf8-614c-4856-8711-5f980c6a6ebe","Type":"ContainerDied","Data":"110c56016716a8074bd49c241cb094558842b86f31aedbe21e73f7fe904067bd"}
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.240278    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9" event={"ID":"5922c0ec-e00e-4d6a-b0a6-08ff53bbb778","Type":"ContainerDied","Data":"ff8e3c7bf6e6b920bd3230795c23ca4c403fb4d5c2e128812c0a6f3d09da72f8"}
Nov 03 01:42:09.241202 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.240288    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"
Nov 03 01:42:09.249951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.249931    4531 scope.go:122] "RemoveContainer" containerID="a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"
Nov 03 01:42:09.250288 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:09.250261    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc\": container with ID starting with a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc not found: ID does not exist" containerID="a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"
Nov 03 01:42:09.250387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.250299    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc"} err="failed to get container status \"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc\": rpc error: code = NotFound desc = could not find container \"a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc\": container with ID starting with a8fbb03d11611883292885f6446c1d174a1e3c34fae133b0a5a7465a343dabdc not found: ID does not exist"
Nov 03 01:42:09.250387 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.250324    4531 scope.go:122] "RemoveContainer" containerID="7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"
Nov 03 01:42:09.262199 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.262155    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:42:09.262402 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.262370    4531 scope.go:122] "RemoveContainer" containerID="7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"
Nov 03 01:42:09.262681 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:09.262659    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127\": container with ID starting with 7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127 not found: ID does not exist" containerID="7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"
Nov 03 01:42:09.262772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.262691    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127"} err="failed to get container status \"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127\": rpc error: code = NotFound desc = could not find container \"7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127\": container with ID starting with 7fd31f0d00de1e10095e6ea87fe185fec33197d4a829930ef1b92f00cbe0d127 not found: ID does not exist"
Nov 03 01:42:09.262772 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.262713    4531 scope.go:122] "RemoveContainer" containerID="d49823d77d876a1431f8c73262e1c689c8f6161b4cbbf7e1f40b54a626562467"
Nov 03 01:42:09.272076 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.272042    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af8244-6c6d7dd874-gnlw9"]
Nov 03 01:42:09.284098 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.284058    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:42:09.285302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285231    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285270    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285287    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285302    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285316    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ms8\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-kube-api-access-v5ms8\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285332    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285348    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285358    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f34cf8-614c-4856-8711-5f980c6a6ebe-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285371    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285390    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285406    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285419    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285433    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285450    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285465    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f34cf8-614c-4856-8711-5f980c6a6ebe-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285481    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjgv\" (UniqueName: \"kubernetes.io/projected/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-kube-api-access-bwjgv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285495    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f34cf8-614c-4856-8711-5f980c6a6ebe-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.285589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.285515    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d4dcef-ba0d-4f3b-8c18-8557274b30e0-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:09.288255 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.288230    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-fjz5c"]
Nov 03 01:42:09.295514 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.295494    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:42:09.299200 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:09.299171    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-lsz2g"]
Nov 03 01:42:09.879651 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:09.879613    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:42:10.905031 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:10.904995    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" path="/var/lib/kubelet/pods/27f34cf8-614c-4856-8711-5f980c6a6ebe/volumes"
Nov 03 01:42:10.905463 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:10.905421    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" path="/var/lib/kubelet/pods/5922c0ec-e00e-4d6a-b0a6-08ff53bbb778/volumes"
Nov 03 01:42:10.905816 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:10.905802    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" path="/var/lib/kubelet/pods/74d4dcef-ba0d-4f3b-8c18-8557274b30e0/volumes"
Nov 03 01:42:17.049735 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.049698    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 01:42:17.050285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.049955    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" podUID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" containerName="maas-api" containerID="cri-o://bdb72984720cb0161b922418099093b426ab09d2f2a3ec35be2b2a657bb74701" gracePeriod=30
Nov 03 01:42:17.315586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.312110    4531 generic.go:356] "Generic (PLEG): container finished" podID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" containerID="bdb72984720cb0161b922418099093b426ab09d2f2a3ec35be2b2a657bb74701" exitCode=0
Nov 03 01:42:17.315586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.312372    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" event={"ID":"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d","Type":"ContainerDied","Data":"bdb72984720cb0161b922418099093b426ab09d2f2a3ec35be2b2a657bb74701"}
Nov 03 01:42:17.396630 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.396604    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 01:42:17.484331 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484299    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\" (UniqueName: \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\") pod \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") "
Nov 03 01:42:17.484545 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484345    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\") pod \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") "
Nov 03 01:42:17.484545 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484405    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") pod \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") "
Nov 03 01:42:17.484545 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\") pod \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\" (UID: \"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d\") "
Nov 03 01:42:17.484770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484737    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca" pod "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" (UID: "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:17.484939 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.484921    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:17.486607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.486584    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd" pod "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" (UID: "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d"). InnerVolumeSpecName "kube-api-access-wrnwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:17.486607 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.486591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs" pod "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" (UID: "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:17.486767 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.486611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls" pod "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" (UID: "c28a6fd1-b43f-45bd-980c-672dc3d6cc1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:17.586322 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.586287    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:17.586322 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.586323    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:17.586545 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:17.586337    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnwd\" (UniqueName: \"kubernetes.io/projected/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d-kube-api-access-wrnwd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:18.320119 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.320089    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"
Nov 03 01:42:18.320592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.320130    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch" event={"ID":"c28a6fd1-b43f-45bd-980c-672dc3d6cc1d","Type":"ContainerDied","Data":"4c39c6194437e0d2239807c07ce8988d8b28faf2a6832718a9344fbdfb10a8c2"}
Nov 03 01:42:18.320592 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.320231    4531 scope.go:122] "RemoveContainer" containerID="bdb72984720cb0161b922418099093b426ab09d2f2a3ec35be2b2a657bb74701"
Nov 03 01:42:18.342693 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.342663    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 01:42:18.351987 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.351953    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-650be0-b78fdb64c-zfkch"]
Nov 03 01:42:18.904447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:18.904413    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" path="/var/lib/kubelet/pods/c28a6fd1-b43f-45bd-980c-672dc3d6cc1d/volumes"
Nov 03 01:42:29.663831 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:29.663789    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 01:42:29.664525 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:29.664046    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy" containerID="cri-o://66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62" gracePeriod=30
Nov 03 01:42:30.391733 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:30.391694    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:42:30.391964 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:30.391937    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy" containerID="cri-o://3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167" gracePeriod=30
Nov 03 01:42:34.962538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:34.962474    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 01:42:35.059807 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059763    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059832    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059889    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059912    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059952    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060008 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.059978    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060035    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060090    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060093    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060138    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\") pod \"e39ee48f-f016-4342-ae50-eb887c944279\" (UID: \"e39ee48f-f016-4342-ae50-eb887c944279\") "
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060225    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.060350 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060266    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.060682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060534    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.060682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060559    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:35.060682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060609    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.060682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060623    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.060682 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.060631    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.062415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.062389    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:35.062415 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.062402    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:35.062657 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.062624    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.062764 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.062723    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj" pod "e39ee48f-f016-4342-ae50-eb887c944279" (UID: "e39ee48f-f016-4342-ae50-eb887c944279"). InnerVolumeSpecName "kube-api-access-4bqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:35.162108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162072    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqwj\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-kube-api-access-4bqwj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.162108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162107    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.162108 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162117    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39ee48f-f016-4342-ae50-eb887c944279-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.162385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162126    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39ee48f-f016-4342-ae50-eb887c944279-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.162385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162136    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39ee48f-f016-4342-ae50-eb887c944279-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.162385 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.162145    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39ee48f-f016-4342-ae50-eb887c944279-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.448913 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.448884    4531 generic.go:356] "Generic (PLEG): container finished" podID="e39ee48f-f016-4342-ae50-eb887c944279" containerID="66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62" exitCode=0
Nov 03 01:42:35.449084 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.448959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" event={"ID":"e39ee48f-f016-4342-ae50-eb887c944279","Type":"ContainerDied","Data":"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"}
Nov 03 01:42:35.449084 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.449006    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"
Nov 03 01:42:35.449084 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.449024    4531 scope.go:122] "RemoveContainer" containerID="66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"
Nov 03 01:42:35.449285 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.449011    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65" event={"ID":"e39ee48f-f016-4342-ae50-eb887c944279","Type":"ContainerDied","Data":"ca4dc557aeeadb011f9324add7b24006487604c37e3a5c586bb1942be666c602"}
Nov 03 01:42:35.486506 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.486479    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 01:42:35.491049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.491027    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-650be0-openshift-default-5c7c5fb7d9-nzh65"]
Nov 03 01:42:35.513453 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.513428    4531 scope.go:122] "RemoveContainer" containerID="66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"
Nov 03 01:42:35.513827 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:35.513804    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62\": container with ID starting with 66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62 not found: ID does not exist" containerID="66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"
Nov 03 01:42:35.513921 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.513834    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62"} err="failed to get container status \"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62\": rpc error: code = NotFound desc = could not find container \"66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62\": container with ID starting with 66f56fc166649f99ae97bb30ea18067d4a674e3838b617d307aa03752c749d62 not found: ID does not exist"
Nov 03 01:42:35.646734 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.646694    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 01:42:35.647771 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.647278    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy" containerID="cri-o://77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32" gracePeriod=30
Nov 03 01:42:35.692846 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.692814    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:42:35.771558 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771095    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.771758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771619    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.771758 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771743    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.771887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771781    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.771887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771833    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.771887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771871    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.772061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771910    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.772061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771952    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.772061 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.771981    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.772317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.772063    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\") pod \"f6d1eaab-3098-4887-ac53-752148bdcfec\" (UID: \"f6d1eaab-3098-4887-ac53-752148bdcfec\") "
Nov 03 01:42:35.772679 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.772652    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.773652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.773094    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.773652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.773317    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.773652 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.773631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:35.774210 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.774186    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.778826 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.776176    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:35.778826 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.777349    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "kube-api-access-qg5j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:35.778826 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.778799    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:35.781622 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.781599    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo" pod "f6d1eaab-3098-4887-ac53-752148bdcfec" (UID: "f6d1eaab-3098-4887-ac53-752148bdcfec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:35.873257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873222    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873251    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873260    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873269    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873277    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873287    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d1eaab-3098-4887-ac53-752148bdcfec-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873296    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg5j6\" (UniqueName: \"kubernetes.io/projected/f6d1eaab-3098-4887-ac53-752148bdcfec-kube-api-access-qg5j6\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:35.873670 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:35.873306    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d1eaab-3098-4887-ac53-752148bdcfec-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:36.461302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.461260    4531 generic.go:356] "Generic (PLEG): container finished" podID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerID="3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167" exitCode=0
Nov 03 01:42:36.461808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.461494    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" event={"ID":"f6d1eaab-3098-4887-ac53-752148bdcfec","Type":"ContainerDied","Data":"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"}
Nov 03 01:42:36.461808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.461552    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg" event={"ID":"f6d1eaab-3098-4887-ac53-752148bdcfec","Type":"ContainerDied","Data":"0c71cce9b9c64b123428b364ff3b7956b55b3bbb3f0676db183d2db1b71f8ccd"}
Nov 03 01:42:36.461808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.461583    4531 scope.go:122] "RemoveContainer" containerID="3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"
Nov 03 01:42:36.461808 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.461684    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"
Nov 03 01:42:36.480275 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.480252    4531 scope.go:122] "RemoveContainer" containerID="3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"
Nov 03 01:42:36.480599 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:36.480578    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167\": container with ID starting with 3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167 not found: ID does not exist" containerID="3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"
Nov 03 01:42:36.480662 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.480612    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167"} err="failed to get container status \"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167\": rpc error: code = NotFound desc = could not find container \"3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167\": container with ID starting with 3f72d95f9999eea08bc0b76cdd708f673a307f9c4f224361c0a1ca35ee0e6167 not found: ID does not exist"
Nov 03 01:42:36.490690 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.490663    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:42:36.499917 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.499891    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2729f2-openshift-default-9fb97b4fc-j28rg"]
Nov 03 01:42:36.903220 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.903190    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39ee48f-f016-4342-ae50-eb887c944279" path="/var/lib/kubelet/pods/e39ee48f-f016-4342-ae50-eb887c944279/volumes"
Nov 03 01:42:36.903683 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:36.903668    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" path="/var/lib/kubelet/pods/f6d1eaab-3098-4887-ac53-752148bdcfec/volumes"
Nov 03 01:42:38.441821 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.441780    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:42:38.442482 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.442445    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" podUID="20a603e3-e890-4171-a0e8-7afbd274bbc5" containerName="maas-api" containerID="cri-o://8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6" gracePeriod=30
Nov 03 01:42:38.739023 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.738996    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:42:38.803930 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.803896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\" (UniqueName: \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\") pod \"20a603e3-e890-4171-a0e8-7afbd274bbc5\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") "
Nov 03 01:42:38.804104 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.804042    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\") pod \"20a603e3-e890-4171-a0e8-7afbd274bbc5\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") "
Nov 03 01:42:38.804148 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.804100    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\") pod \"20a603e3-e890-4171-a0e8-7afbd274bbc5\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") "
Nov 03 01:42:38.804228 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.804211    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\") pod \"20a603e3-e890-4171-a0e8-7afbd274bbc5\" (UID: \"20a603e3-e890-4171-a0e8-7afbd274bbc5\") "
Nov 03 01:42:38.804509 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.804478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca" pod "20a603e3-e890-4171-a0e8-7afbd274bbc5" (UID: "20a603e3-e890-4171-a0e8-7afbd274bbc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:38.804644 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.804590    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a603e3-e890-4171-a0e8-7afbd274bbc5-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:38.806043 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.806017    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4" pod "20a603e3-e890-4171-a0e8-7afbd274bbc5" (UID: "20a603e3-e890-4171-a0e8-7afbd274bbc5"). InnerVolumeSpecName "kube-api-access-6mlw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:38.806224 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.806201    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls" pod "20a603e3-e890-4171-a0e8-7afbd274bbc5" (UID: "20a603e3-e890-4171-a0e8-7afbd274bbc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:38.806340 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.806321    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs" pod "20a603e3-e890-4171-a0e8-7afbd274bbc5" (UID: "20a603e3-e890-4171-a0e8-7afbd274bbc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:38.905417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.905380    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:38.905417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.905417    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a603e3-e890-4171-a0e8-7afbd274bbc5-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:38.905646 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:38.905433    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlw4\" (UniqueName: \"kubernetes.io/projected/20a603e3-e890-4171-a0e8-7afbd274bbc5-kube-api-access-6mlw4\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:39.495216 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.495179    4531 generic.go:356] "Generic (PLEG): container finished" podID="20a603e3-e890-4171-a0e8-7afbd274bbc5" containerID="8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6" exitCode=0
Nov 03 01:42:39.495724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.495229    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"
Nov 03 01:42:39.495724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.495251    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" event={"ID":"20a603e3-e890-4171-a0e8-7afbd274bbc5","Type":"ContainerDied","Data":"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"}
Nov 03 01:42:39.495724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.495309    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj" event={"ID":"20a603e3-e890-4171-a0e8-7afbd274bbc5","Type":"ContainerDied","Data":"668a5c55271d4d1af31f274d912ea69fec91ab210f9827a888992a4574f6c9d8"}
Nov 03 01:42:39.495724 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.495335    4531 scope.go:122] "RemoveContainer" containerID="8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"
Nov 03 01:42:39.505067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.505047    4531 scope.go:122] "RemoveContainer" containerID="8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"
Nov 03 01:42:39.505341 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:39.505323    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6\": container with ID starting with 8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6 not found: ID does not exist" containerID="8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"
Nov 03 01:42:39.505399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.505365    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6"} err="failed to get container status \"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6\": rpc error: code = NotFound desc = could not find container \"8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6\": container with ID starting with 8c6713491c8d508c557353861b872922b7b2d5843e2b41b26eef36584f5e18d6 not found: ID does not exist"
Nov 03 01:42:39.517673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.515694    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:42:39.519635 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:39.519604    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-4af1f7-7d6f556fc-tx6jj"]
Nov 03 01:42:39.879581 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:39.879540    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:42:40.902595 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:40.902545    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a603e3-e890-4171-a0e8-7afbd274bbc5" path="/var/lib/kubelet/pods/20a603e3-e890-4171-a0e8-7afbd274bbc5/volumes"
Nov 03 01:42:41.342794 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.342766    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 01:42:41.455231 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455106    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455264    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455316    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455381    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455420 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455391    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:41.455663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455438    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455521    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455663 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455616    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455665    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\") pod \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\" (UID: \"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e\") "
Nov 03 01:42:41.455892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455668    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:41.455892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455745    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:41.455892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:41.455892 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.455875    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:41.456257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.456066    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.456257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.456080    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.456257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.456088    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.456257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.456099    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.456257 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.456116    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.457527 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.457501    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "kube-api-access-fnsvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:41.457702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.457680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:41.457777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.457685    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:42:41.457859 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.457836    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo" pod "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" (UID: "d5a10afe-dd1f-4701-b8f9-ace98dee8b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:42:41.512495 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.512446    4531 generic.go:356] "Generic (PLEG): container finished" podID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerID="77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32" exitCode=0
Nov 03 01:42:41.512664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.512593    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" event={"ID":"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e","Type":"ContainerDied","Data":"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"}
Nov 03 01:42:41.512664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.512646    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w" event={"ID":"d5a10afe-dd1f-4701-b8f9-ace98dee8b5e","Type":"ContainerDied","Data":"e51efe69547a314556f148e26cfeb667ccf2be6188093e1b9709393d291389e2"}
Nov 03 01:42:41.512792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.512674    4531 scope.go:122] "RemoveContainer" containerID="77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"
Nov 03 01:42:41.512792 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.512781    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"
Nov 03 01:42:41.528391 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.528355    4531 scope.go:122] "RemoveContainer" containerID="77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"
Nov 03 01:42:41.528678 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:42:41.528660    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32\": container with ID starting with 77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32 not found: ID does not exist" containerID="77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"
Nov 03 01:42:41.528752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.528685    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32"} err="failed to get container status \"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32\": rpc error: code = NotFound desc = could not find container \"77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32\": container with ID starting with 77a9447e357e138ff26c641191210f28b70373bfe0e57e0afb13fa3f4c6b5d32 not found: ID does not exist"
Nov 03 01:42:41.538417 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.538390    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 01:42:41.543009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.542987    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-83755a-openshift-default-c86f9565c-64j8w"]
Nov 03 01:42:41.556673 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.556650    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.556777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.556675    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.556777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.556691    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:41.556777 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:41.556704    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsvr\" (UniqueName: \"kubernetes.io/projected/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e-kube-api-access-fnsvr\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:42.902727 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:42.902691    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" path="/var/lib/kubelet/pods/d5a10afe-dd1f-4701-b8f9-ace98dee8b5e/volumes"
Nov 03 01:42:50.149925 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:50.149883    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 01:42:50.159175 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:42:50.159127    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 01:43:05.174871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.174826    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main" containerID="cri-o://7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98" gracePeriod=45
Nov 03 01:43:05.187631 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.187592    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main" containerID="cri-o://0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099" gracePeriod=45
Nov 03 01:43:05.453933 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.453904    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 01:43:05.497295 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.497261    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:43:05.522260 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522222    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.522477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522294    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\" (UniqueName: \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.522477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522331    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522358    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.522477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522427    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\" (UniqueName: \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522477 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522474    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522540    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522571    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522620    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522654    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\") pod \"69203221-8dae-4a89-b636-321a7a2442f8\" (UID: \"69203221-8dae-4a89-b636-321a7a2442f8\") "
Nov 03 01:43:05.522765 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.522725    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\") pod \"9f203175-33d7-44d8-9062-e5e592473a97\" (UID: \"9f203175-33d7-44d8-9062-e5e592473a97\") "
Nov 03 01:43:05.523215 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.523186    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.523392 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.523370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.523474 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.523438    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.523881 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.523855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.526460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.526383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:43:05.526460 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.526423    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.526919 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.526896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.527009 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.526984    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.527134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.527116    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.527546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.527141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "kube-api-access-x5mwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:43:05.527666 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.527546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:43:05.529779 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.529752    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn" pod "69203221-8dae-4a89-b636-321a7a2442f8" (UID: "69203221-8dae-4a89-b636-321a7a2442f8"). InnerVolumeSpecName "kube-api-access-98ljn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:43:05.530634 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.530609    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm" pod "9f203175-33d7-44d8-9062-e5e592473a97" (UID: "9f203175-33d7-44d8-9062-e5e592473a97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:05.624694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624654    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5mwx\" (UniqueName: \"kubernetes.io/projected/9f203175-33d7-44d8-9062-e5e592473a97-kube-api-access-x5mwx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624692    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624703    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624713    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ljn\" (UniqueName: \"kubernetes.io/projected/69203221-8dae-4a89-b636-321a7a2442f8-kube-api-access-98ljn\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624722    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f203175-33d7-44d8-9062-e5e592473a97-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624730    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-home\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624739    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624747    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-model-cache\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624754    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69203221-8dae-4a89-b636-321a7a2442f8-tls-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624762    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69203221-8dae-4a89-b636-321a7a2442f8-tmp-dir\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.624951 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.624771    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f203175-33d7-44d8-9062-e5e592473a97-dshm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:05.670195 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.670141    4531 generic.go:356] "Generic (PLEG): container finished" podID="9f203175-33d7-44d8-9062-e5e592473a97" containerID="7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98" exitCode=0
Nov 03 01:43:05.670399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.670202    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"
Nov 03 01:43:05.670399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.670230    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" event={"ID":"9f203175-33d7-44d8-9062-e5e592473a97","Type":"ContainerDied","Data":"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"}
Nov 03 01:43:05.670399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.670273    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm" event={"ID":"9f203175-33d7-44d8-9062-e5e592473a97","Type":"ContainerDied","Data":"8385b5ff185ddece9966e992ef9a63f8b115054cc1664904cdc78070aa2e78c9"}
Nov 03 01:43:05.670399 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.670292    4531 scope.go:122] "RemoveContainer" containerID="7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"
Nov 03 01:43:05.671798 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.671775    4531 generic.go:356] "Generic (PLEG): container finished" podID="69203221-8dae-4a89-b636-321a7a2442f8" containerID="0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099" exitCode=0
Nov 03 01:43:05.671938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.671833    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"
Nov 03 01:43:05.671938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.671856    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" event={"ID":"69203221-8dae-4a89-b636-321a7a2442f8","Type":"ContainerDied","Data":"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"}
Nov 03 01:43:05.671938 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.671886    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7" event={"ID":"69203221-8dae-4a89-b636-321a7a2442f8","Type":"ContainerDied","Data":"ca2b87a94bc3e37e637488c16f096d1b78b345c9978d0e79161b08fb5267f659"}
Nov 03 01:43:05.685041 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.685017    4531 scope.go:122] "RemoveContainer" containerID="7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"
Nov 03 01:43:05.685787 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:43:05.685726    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98\": container with ID starting with 7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98 not found: ID does not exist" containerID="7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"
Nov 03 01:43:05.685787 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.685765    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98"} err="failed to get container status \"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98\": rpc error: code = NotFound desc = could not find container \"7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98\": container with ID starting with 7eef7ea42e353e55c98d597d81676439d50cdead4efc3b98b308de505677ec98 not found: ID does not exist"
Nov 03 01:43:05.685937 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.685790    4531 scope.go:122] "RemoveContainer" containerID="0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"
Nov 03 01:43:05.694513 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.694492    4531 scope.go:122] "RemoveContainer" containerID="0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"
Nov 03 01:43:05.694809 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:43:05.694789    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099\": container with ID starting with 0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099 not found: ID does not exist" containerID="0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"
Nov 03 01:43:05.694885 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.694817    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099"} err="failed to get container status \"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099\": rpc error: code = NotFound desc = could not find container \"0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099\": container with ID starting with 0ace735d53ab9aceea2b30b9f5e924273cbe6b39a88988be38d9c086213e9099 not found: ID does not exist"
Nov 03 01:43:05.698390 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.698364    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 01:43:05.701668 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.701642    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/distinct-w1-88bbf7-kserve-64754949f9-jlvvm"]
Nov 03 01:43:05.712531 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.712508    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 01:43:05.714636 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:05.714616    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-88bbf7/premium-w1-88bbf7-kserve-75b7cbcd5b-x25m7"]
Nov 03 01:43:06.902609 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:06.902534    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69203221-8dae-4a89-b636-321a7a2442f8" path="/var/lib/kubelet/pods/69203221-8dae-4a89-b636-321a7a2442f8/volumes"
Nov 03 01:43:06.902978 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:06.902951    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f203175-33d7-44d8-9062-e5e592473a97" path="/var/lib/kubelet/pods/9f203175-33d7-44d8-9062-e5e592473a97/volumes"
Nov 03 01:43:07.527627 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.527592    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 01:43:07.527869 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.527842    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" podUID="06a74844-b9b3-4916-9bbb-f39cfc670094" containerName="maas-api" containerID="cri-o://db580f8424ed51a4538da55f04a1d1dcb6e2a288a713d1c4d37ccf6b5fb47f04" gracePeriod=30
Nov 03 01:43:07.688467 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.688434    4531 generic.go:356] "Generic (PLEG): container finished" podID="06a74844-b9b3-4916-9bbb-f39cfc670094" containerID="db580f8424ed51a4538da55f04a1d1dcb6e2a288a713d1c4d37ccf6b5fb47f04" exitCode=0
Nov 03 01:43:07.688662 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.688493    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" event={"ID":"06a74844-b9b3-4916-9bbb-f39cfc670094","Type":"ContainerDied","Data":"db580f8424ed51a4538da55f04a1d1dcb6e2a288a713d1c4d37ccf6b5fb47f04"}
Nov 03 01:43:07.819274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.819245    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 01:43:07.848380 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.848344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\") pod \"06a74844-b9b3-4916-9bbb-f39cfc670094\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") "
Nov 03 01:43:07.848563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.848467    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\") pod \"06a74844-b9b3-4916-9bbb-f39cfc670094\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") "
Nov 03 01:43:07.848563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.848512    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\" (UniqueName: \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\") pod \"06a74844-b9b3-4916-9bbb-f39cfc670094\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") "
Nov 03 01:43:07.848563 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.848539    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\") pod \"06a74844-b9b3-4916-9bbb-f39cfc670094\" (UID: \"06a74844-b9b3-4916-9bbb-f39cfc670094\") "
Nov 03 01:43:07.849050 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.849020    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca" pod "06a74844-b9b3-4916-9bbb-f39cfc670094" (UID: "06a74844-b9b3-4916-9bbb-f39cfc670094"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:43:07.851178 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.851088    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs" pod "06a74844-b9b3-4916-9bbb-f39cfc670094" (UID: "06a74844-b9b3-4916-9bbb-f39cfc670094"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:43:07.851178 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.851128    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls" pod "06a74844-b9b3-4916-9bbb-f39cfc670094" (UID: "06a74844-b9b3-4916-9bbb-f39cfc670094"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:43:07.851368 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.851215    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn" pod "06a74844-b9b3-4916-9bbb-f39cfc670094" (UID: "06a74844-b9b3-4916-9bbb-f39cfc670094"). InnerVolumeSpecName "kube-api-access-vtpsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:43:07.950003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.949966    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:07.950003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.949996    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a74844-b9b3-4916-9bbb-f39cfc670094-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:07.950003 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.950009    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtpsn\" (UniqueName: \"kubernetes.io/projected/06a74844-b9b3-4916-9bbb-f39cfc670094-kube-api-access-vtpsn\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:07.950638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:07.950018    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a74844-b9b3-4916-9bbb-f39cfc670094-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:08.697049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.697010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w" event={"ID":"06a74844-b9b3-4916-9bbb-f39cfc670094","Type":"ContainerDied","Data":"b7272737b7f29019a69a898b3511e5c9c55aacc0da72d0b577e4d63a9991b1d3"}
Nov 03 01:43:08.697289 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.697115    4531 scope.go:122] "RemoveContainer" containerID="db580f8424ed51a4538da55f04a1d1dcb6e2a288a713d1c4d37ccf6b5fb47f04"
Nov 03 01:43:08.697383 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.697334    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"
Nov 03 01:43:08.726841 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.726807    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 01:43:08.731081 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.731044    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-88bbf7-7d464c5647-64m2w"]
Nov 03 01:43:08.901839 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:08.901803    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a74844-b9b3-4916-9bbb-f39cfc670094" path="/var/lib/kubelet/pods/06a74844-b9b3-4916-9bbb-f39cfc670094/volumes"
Nov 03 01:43:09.879933 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:43:09.879892    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:43:29.248868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:29.248824    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 01:43:29.818427 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:29.818384    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" containerID="cri-o://fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e" gracePeriod=30
Nov 03 01:43:30.564229 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:30.564186    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c9svn_d65cb9f3-5969-41c3-8f50-42bf0f002c72/manager/0.log"
Nov 03 01:43:30.760266 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:30.760233    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/1.log"
Nov 03 01:43:31.533561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:31.533522    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7zsk9_78d9bc3e-fee0-499f-9c05-f6fcdd169cfd/manager/0.log"
Nov 03 01:43:31.977176 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:31.977131    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-szqjl_dbf70c07-8ae7-4884-933c-17d7dccf6806/manager/0.log"
Nov 03 01:43:32.561087 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:32.561057    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/0.log"
Nov 03 01:43:33.293133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:33.293104    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/0.log"
Nov 03 01:43:33.478092 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:33.478062    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-n922j_29693553-4e1c-4e36-aad0-a1360b84ba72/manager/0.log"
Nov 03 01:43:33.691588 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:33.691555    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kkdsj_8d36a864-53d0-4e75-b3bc-867fc1d42a89/manager/0.log"
Nov 03 01:43:34.293638 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:34.293606    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:43:34.296018 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:34.293668    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:43:34.394933 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:34.394904    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knbqk_49fee83f-2189-40b6-8559-022d6cc91770/server/0.log"
Nov 03 01:43:34.573932 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:34.573908    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ctxbc_11772d42-58e7-41ad-8adb-28220a88c2f9/manager/0.log"
Nov 03 01:43:34.778221 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:34.778182    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzws2_2432b11a-588d-42c8-9698-1787ca42ed40/proxy/0.log"
Nov 03 01:43:35.074561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.074534    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 01:43:35.237156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237060    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237116    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237156 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237146    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237223    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237267    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237322    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237379    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237422    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237458    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\") pod \"d4c35222-39b0-405f-8248-747f49e20090\" (UID: \"d4c35222-39b0-405f-8248-747f49e20090\") "
Nov 03 01:43:35.237502 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237456    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:35.237887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237543    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:35.237887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237622    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:35.237887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237638    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:35.237887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237745    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:43:35.237887 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237885    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-data\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.238096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237902    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c35222-39b0-405f-8248-747f49e20090-istiod-ca-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.238096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237926    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.238096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237944    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-workload-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.238096 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.237958    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-credential-socket\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.239589 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.239562    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:43:35.239716 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.239630    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "kube-api-access-drkp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:43:35.239781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.239764    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:43:35.239832 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.239816    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo" pod "d4c35222-39b0-405f-8248-747f49e20090" (UID: "d4c35222-39b0-405f-8248-747f49e20090"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:43:35.339066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.339033    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c35222-39b0-405f-8248-747f49e20090-istio-podinfo\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.339066 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.339063    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c35222-39b0-405f-8248-747f49e20090-istio-envoy\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.339538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.339076    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drkp5\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-kube-api-access-drkp5\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.339538 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.339086    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c35222-39b0-405f-8248-747f49e20090-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:43:35.856511 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.856473    4531 generic.go:356] "Generic (PLEG): container finished" podID="d4c35222-39b0-405f-8248-747f49e20090" containerID="fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e" exitCode=0
Nov 03 01:43:35.856696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.856546    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"
Nov 03 01:43:35.856696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.856563    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" event={"ID":"d4c35222-39b0-405f-8248-747f49e20090","Type":"ContainerDied","Data":"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"}
Nov 03 01:43:35.856696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.856611    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l" event={"ID":"d4c35222-39b0-405f-8248-747f49e20090","Type":"ContainerDied","Data":"75afa3fd91351ae1f588a2e13ea39752e8777f96ad6b1a8147ba91075fd7901e"}
Nov 03 01:43:35.856696 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.856635    4531 scope.go:122] "RemoveContainer" containerID="fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"
Nov 03 01:43:35.866608 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.866588    4531 scope.go:122] "RemoveContainer" containerID="fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"
Nov 03 01:43:35.866885 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:43:35.866864    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e\": container with ID starting with fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e not found: ID does not exist" containerID="fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"
Nov 03 01:43:35.866956 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.866892    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e"} err="failed to get container status \"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e\": rpc error: code = NotFound desc = could not find container \"fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e\": container with ID starting with fba7da635276ce193d4b165a44be3649c5492e4727c203ab735fa5f5ac0d575e not found: ID does not exist"
Nov 03 01:43:35.879004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.878979    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 01:43:35.882901 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:35.882881    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-88bbf7-openshift-default-574f47fff7-sbh4l"]
Nov 03 01:43:36.906288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:43:36.906255    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c35222-39b0-405f-8248-747f49e20090" path="/var/lib/kubelet/pods/d4c35222-39b0-405f-8248-747f49e20090/volumes"
Nov 03 01:43:39.883575 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:43:39.883527    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:44:09.881114 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:44:09.881069    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:44:39.876669 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:44:39.876586    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:44:49.012212 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012150    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"]
Nov 03 01:44:49.012837 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012816    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012839    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012860    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012868    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012886    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012895    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012904    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy"
Nov 03 01:44:49.012915 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012913    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012927    4531 cpu_manager.go:414] "containerMap: removing container" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012934    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012947    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012954    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012963    4531 cpu_manager.go:414] "containerMap: removing container" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012970    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012983    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.012990    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013000    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013007    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013016    4531 cpu_manager.go:414] "containerMap: removing container" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013024    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013034    4531 cpu_manager.go:414] "containerMap: removing container" podUID="00535132-f427-428e-9b3d-301eeace95db" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013040    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00535132-f427-428e-9b3d-301eeace95db" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013052    4531 cpu_manager.go:414] "containerMap: removing container" podUID="06a74844-b9b3-4916-9bbb-f39cfc670094" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013060    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a74844-b9b3-4916-9bbb-f39cfc670094" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013068    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013074    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013088    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013095    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013106    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013113    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013124    4531 cpu_manager.go:414] "containerMap: removing container" podUID="20a603e3-e890-4171-a0e8-7afbd274bbc5" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013131    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a603e3-e890-4171-a0e8-7afbd274bbc5" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013145    4531 cpu_manager.go:414] "containerMap: removing container" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013150    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013174    4531 cpu_manager.go:414] "containerMap: removing container" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013180    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013191    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013196    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013203    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013207    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013213    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3865d725-5e1d-4dd6-ab4c-071637eadfad" containerName="maas-api"
Nov 03 01:44:49.013274 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013217    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3865d725-5e1d-4dd6-ab4c-071637eadfad" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013304    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d4dcef-ba0d-4f3b-8c18-8557274b30e0" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013314    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f203175-33d7-44d8-9062-e5e592473a97" containerName="main"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013319    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c35222-39b0-405f-8248-747f49e20090" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013324    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3865d725-5e1d-4dd6-ab4c-071637eadfad" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013333    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a10afe-dd1f-4701-b8f9-ace98dee8b5e" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013339    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39ee48f-f016-4342-ae50-eb887c944279" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013346    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d1eaab-3098-4887-ac53-752148bdcfec" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013352    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f631a-2510-4c8e-9f9d-791087aa43e1" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013357    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="714ac1f7-1b14-4755-b1dc-5278b5df6ced" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013361    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5922c0ec-e00e-4d6a-b0a6-08ff53bbb778" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013366    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f34cf8-614c-4856-8711-5f980c6a6ebe" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013373    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1aeaf6-086e-49dc-900c-18870ba9ae0c" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013379    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f037a05a-daaf-4185-9405-2fd5e4112f6a" containerName="main"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013384    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="18770058-d434-4507-84e5-e7a64fdccb03" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013389    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00535132-f427-428e-9b3d-301eeace95db" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013395    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a74844-b9b3-4916-9bbb-f39cfc670094" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013400    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28a6fd1-b43f-45bd-980c-672dc3d6cc1d" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013406    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21dd413-d51f-4eaf-9f47-12eab1686a34" containerName="istio-proxy"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013411    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e524439-cc34-4594-8570-64ba80a8ea8d" containerName="main"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013416    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a603e3-e890-4171-a0e8-7afbd274bbc5" containerName="maas-api"
Nov 03 01:44:49.014781 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.013422    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="69203221-8dae-4a89-b636-321a7a2442f8" containerName="main"
Nov 03 01:44:49.017105 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.017070    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.032325 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.032291    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"]
Nov 03 01:44:49.188695 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.188653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ac183a-e533-4d5f-91c4-94d05308d76a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.188889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.188728    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.188889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.188819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.188889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.188849    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.188889 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.188881    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.189049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.189001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khlj\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-kube-api-access-6khlj\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.189049 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.189028    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.189153 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.189050    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.189238 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.189145    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290041    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290134 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290100    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khlj\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-kube-api-access-6khlj\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290183    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290207    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290246    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ac183a-e533-4d5f-91c4-94d05308d76a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290458 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290340    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290645    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-data\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290770 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-credential-socket\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.290875 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.290828    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-workload-certs\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.291133 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.291113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ac183a-e533-4d5f-91c4-94d05308d76a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.292388 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.292368    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-envoy\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.292677 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.292656    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.298490 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.298464    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-istio-token\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.298586 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.298555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khlj\" (UniqueName: \"kubernetes.io/projected/51ac183a-e533-4d5f-91c4-94d05308d76a-kube-api-access-6khlj\") pod \"maas-default-gateway-openshift-default-596dc9545f-ptjsj\" (UID: \"51ac183a-e533-4d5f-91c4-94d05308d76a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.334067 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.334041    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:49.615598 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.615568    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"]
Nov 03 01:44:49.617520 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:44:49.617490    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ac183a_e533_4d5f_91c4_94d05308d76a.slice/crio-f781246226e69925091f3ffff27da472e7ce18be73fa7445c646c5f80a84ad20 WatchSource:0}: Error finding container f781246226e69925091f3ffff27da472e7ce18be73fa7445c646c5f80a84ad20: Status 404 returned error can't find the container with id f781246226e69925091f3ffff27da472e7ce18be73fa7445c646c5f80a84ad20
Nov 03 01:44:49.619728 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.619696    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:44:49.619825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.619759    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:44:49.619825 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:49.619788    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:44:50.276937 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.276901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" event={"ID":"51ac183a-e533-4d5f-91c4-94d05308d76a","Type":"ContainerStarted","Data":"201e07dc9804c440ebe0ad8fe85bb226430b5b143b39c199b8375fedc5b91b42"}
Nov 03 01:44:50.277450 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.276957    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" event={"ID":"51ac183a-e533-4d5f-91c4-94d05308d76a","Type":"ContainerStarted","Data":"f781246226e69925091f3ffff27da472e7ce18be73fa7445c646c5f80a84ad20"}
Nov 03 01:44:50.298745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.298698    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podStartSLOduration=2.298683766 podStartE2EDuration="2.298683766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:50.295051539 +0000 UTC m=+3835.508795212" watchObservedRunningTime="2026-11-03 01:44:50.298683766 +0000 UTC m=+3835.512427495"
Nov 03 01:44:50.351433 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.351395    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:50.353026 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.352997    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:50.353184 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:50.353047    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:51.346709 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:51.346663    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:51.347335 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:51.347298    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:52.350327 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:52.350283    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:52.350958 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:52.350924    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:53.344190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:53.341553    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:53.344190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:53.341635    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:54.344190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:54.334540    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:54.344190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:54.334616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:55.349304 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:55.349261    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:55.349918 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:55.349876    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:56.334929 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:56.334893    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-ptjsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 01:44:56.335116 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:56.334943    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj" podUID="51ac183a-e533-4d5f-91c4-94d05308d76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 01:44:57.802201 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:57.802150    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:57.802742 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:57.802618    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:44:57.803585 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:44:57.803565    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-ptjsj"
Nov 03 01:45:09.878375 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:45:09.878336    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:45:16.349726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:45:16.349257    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:45:16.349726 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:45:16.349347    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:45:39.878268 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:45:39.878225    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:45:59.721871 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:45:59.721826    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 01:45:59.722471 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:45:59.722149    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerName="discovery" containerID="cri-o://dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c" gracePeriod=30
Nov 03 01:45:59.997447 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:45:59.997420    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 01:46:00.069055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069008    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069073    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069271 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069187    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069286    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069325    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069505 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069438    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\") pod \"083bc720-a6ca-480f-ba2d-14f88600f1f4\" (UID: \"083bc720-a6ca-480f-ba2d-14f88600f1f4\") "
Nov 03 01:46:00.069694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069493    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:00.069819 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.069794    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-ca-configmap\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.071975 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.071942    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:46:00.072337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.072301    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:00.072459 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.072415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:00.072550 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.072518    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "kube-api-access-ln8dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:00.072621 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.072524    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:46:00.072720 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.072697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert" pod "083bc720-a6ca-480f-ba2d-14f88600f1f4" (UID: "083bc720-a6ca-480f-ba2d-14f88600f1f4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:46:00.170834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170787    4531 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-kubeconfig\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.170834 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170832    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-token\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.171058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170850    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8dd\" (UniqueName: \"kubernetes.io/projected/083bc720-a6ca-480f-ba2d-14f88600f1f4-kube-api-access-ln8dd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.171058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170865    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-istio-csr-dns-cert\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.171058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170878    4531 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/083bc720-a6ca-480f-ba2d-14f88600f1f4-cacerts\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.171058 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.170892    4531 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/083bc720-a6ca-480f-ba2d-14f88600f1f4-local-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:00.748117 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.748079    4531 generic.go:356] "Generic (PLEG): container finished" podID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerID="dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c" exitCode=0
Nov 03 01:46:00.748637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.748127    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"
Nov 03 01:46:00.748637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.748184    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" event={"ID":"083bc720-a6ca-480f-ba2d-14f88600f1f4","Type":"ContainerDied","Data":"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"}
Nov 03 01:46:00.748637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.748249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8" event={"ID":"083bc720-a6ca-480f-ba2d-14f88600f1f4","Type":"ContainerDied","Data":"6bca525c1c8e49758db6b1a63721dbe8cd2326e0250b4d9969bc27c9325498f6"}
Nov 03 01:46:00.748637 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.748278    4531 scope.go:122] "RemoveContainer" containerID="dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"
Nov 03 01:46:00.765302 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.765280    4531 scope.go:122] "RemoveContainer" containerID="dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"
Nov 03 01:46:00.765579 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:46:00.765554    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c\": container with ID starting with dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c not found: ID does not exist" containerID="dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"
Nov 03 01:46:00.765629 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.765588    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c"} err="failed to get container status \"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c\": rpc error: code = NotFound desc = could not find container \"dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c\": container with ID starting with dd13dca5b63e75e1bce540de8f6f9ed90bdea12fc35613c588f5f82d9ae7831c not found: ID does not exist"
Nov 03 01:46:00.771132 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.771106    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 01:46:00.779606 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.779582    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mktb8"]
Nov 03 01:46:00.903731 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:00.903685    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" path="/var/lib/kubelet/pods/083bc720-a6ca-480f-ba2d-14f88600f1f4/volumes"
Nov 03 01:46:09.877187 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:46:09.877085    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:46:23.810185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:23.809420    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:46:23.810185 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:46:23.809524    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:46:39.877457 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:46:39.877422    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:47:09.876491 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:47:09.876452    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:47:39.879960 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:47:39.879882    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:47:42.342288 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.342250    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:47:42.342942 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.342920    4531 cpu_manager.go:414] "containerMap: removing container" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerName="discovery"
Nov 03 01:47:42.343036 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.342945    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerName="discovery"
Nov 03 01:47:42.343099 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.343080    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="083bc720-a6ca-480f-ba2d-14f88600f1f4" containerName="discovery"
Nov 03 01:47:42.346856 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.346834    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.353865 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.353828    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:47:42.355511 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.355487    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-515c8f-serving-cert\""
Nov 03 01:47:42.427694 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.427657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xc5\" (UniqueName: \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.427868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.427718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.427868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.427824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.427868 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.427856    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.529431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.529399    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.529431 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.529432    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.529658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.529501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xc5\" (UniqueName: \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.529658 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.529527    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.530205 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.530150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.531906 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.531870    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.532024 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.531959    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.544546 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.544519    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xc5\" (UniqueName: \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\") pod \"maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.676708 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.676622    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:42.953142 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.953113    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:47:42.955326 ip-10-0-136-112 kubenswrapper[4531]: W1103 01:47:42.955288    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cda9d8_ab58_4714_a79a_70e7647474fc.slice/crio-43766641576df12e4fa7c1d3343ee8f7b1649d10bb6b4874d3467e47df7e6210 WatchSource:0}: Error finding container 43766641576df12e4fa7c1d3343ee8f7b1649d10bb6b4874d3467e47df7e6210: Status 404 returned error can't find the container with id 43766641576df12e4fa7c1d3343ee8f7b1649d10bb6b4874d3467e47df7e6210
Nov 03 01:47:42.956671 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:42.956650    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:47:43.462146 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:43.462109    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" event={"ID":"c0cda9d8-ab58-4714-a79a-70e7647474fc","Type":"ContainerStarted","Data":"43766641576df12e4fa7c1d3343ee8f7b1649d10bb6b4874d3467e47df7e6210"}
Nov 03 01:47:44.469321 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:44.469286    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" event={"ID":"c0cda9d8-ab58-4714-a79a-70e7647474fc","Type":"ContainerStarted","Data":"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"}
Nov 03 01:47:44.469754 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:44.469428    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:47:44.491021 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:44.490905    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" podStartSLOduration=1.935787669 podStartE2EDuration="2.490883935s" totalImagesPullingTime="555.096266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:44.488846947 +0000 UTC m=+4009.702590620" watchObservedRunningTime="2026-11-03 01:47:44.490883935 +0000 UTC m=+4009.704627608"
Nov 03 01:47:50.481745 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:47:50.481713    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:48:09.877254 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:48:09.877216    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:48:39.877170 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:48:39.877117    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:48:50.004190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.003417    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:48:50.004190 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.003723    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" podUID="c0cda9d8-ab58-4714-a79a-70e7647474fc" containerName="maas-api" containerID="cri-o://fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a" gracePeriod=30
Nov 03 01:48:50.282317 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.282289    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:48:50.350320 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.350289    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\") pod \"c0cda9d8-ab58-4714-a79a-70e7647474fc\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") "
Nov 03 01:48:50.350519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.350415    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\" (UniqueName: \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\") pod \"c0cda9d8-ab58-4714-a79a-70e7647474fc\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") "
Nov 03 01:48:50.350519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.350442    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\") pod \"c0cda9d8-ab58-4714-a79a-70e7647474fc\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") "
Nov 03 01:48:50.350519 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.350482    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\") pod \"c0cda9d8-ab58-4714-a79a-70e7647474fc\" (UID: \"c0cda9d8-ab58-4714-a79a-70e7647474fc\") "
Nov 03 01:48:50.350979 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.350942    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca" pod "c0cda9d8-ab58-4714-a79a-70e7647474fc" (UID: "c0cda9d8-ab58-4714-a79a-70e7647474fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:50.352451 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.352421    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs" pod "c0cda9d8-ab58-4714-a79a-70e7647474fc" (UID: "c0cda9d8-ab58-4714-a79a-70e7647474fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:48:50.352581 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.352565    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls" pod "c0cda9d8-ab58-4714-a79a-70e7647474fc" (UID: "c0cda9d8-ab58-4714-a79a-70e7647474fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:48:50.352702 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.352680    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5" pod "c0cda9d8-ab58-4714-a79a-70e7647474fc" (UID: "c0cda9d8-ab58-4714-a79a-70e7647474fc"). InnerVolumeSpecName "kube-api-access-x8xc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:50.451842 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.451808    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:50.451842 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.451839    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cda9d8-ab58-4714-a79a-70e7647474fc-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:50.451842 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.451851    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cda9d8-ab58-4714-a79a-70e7647474fc-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:50.452091 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.451860    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8xc5\" (UniqueName: \"kubernetes.io/projected/c0cda9d8-ab58-4714-a79a-70e7647474fc-kube-api-access-x8xc5\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:50.858440 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.858407    4531 generic.go:356] "Generic (PLEG): container finished" podID="c0cda9d8-ab58-4714-a79a-70e7647474fc" containerID="fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a" exitCode=0
Nov 03 01:48:50.858664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.858456    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"
Nov 03 01:48:50.858664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.858468    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" event={"ID":"c0cda9d8-ab58-4714-a79a-70e7647474fc","Type":"ContainerDied","Data":"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"}
Nov 03 01:48:50.858664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.858511    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9" event={"ID":"c0cda9d8-ab58-4714-a79a-70e7647474fc","Type":"ContainerDied","Data":"43766641576df12e4fa7c1d3343ee8f7b1649d10bb6b4874d3467e47df7e6210"}
Nov 03 01:48:50.858664 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.858532    4531 scope.go:122] "RemoveContainer" containerID="fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"
Nov 03 01:48:50.868628 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.868610    4531 scope.go:122] "RemoveContainer" containerID="fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"
Nov 03 01:48:50.868968 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:48:50.868938    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a\": container with ID starting with fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a not found: ID does not exist" containerID="fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"
Nov 03 01:48:50.869055 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.868978    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a"} err="failed to get container status \"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a\": rpc error: code = NotFound desc = could not find container \"fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a\": container with ID starting with fc8c50f714c2efa18c962d6f3804e4406a0bb8c188510d90fded1f785a7eb24a not found: ID does not exist"
Nov 03 01:48:50.881337 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.881307    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:48:50.885752 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.885727    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-515c8f-6bb9bf56f4-grzp9"]
Nov 03 01:48:50.903561 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:48:50.903533    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cda9d8-ab58-4714-a79a-70e7647474fc" path="/var/lib/kubelet/pods/c0cda9d8-ab58-4714-a79a-70e7647474fc/volumes"
Nov 03 01:49:09.877936 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:49:09.877851    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:49:39.878808 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:49:39.878770    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:50:09.876974 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:50:09.876934    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:50:39.876893 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:50:39.876798    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:51:09.876829 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:51:09.876791    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:51:39.876678 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:51:39.876635    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:52:09.877065 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:52:09.876985    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:52:39.876671 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:52:39.876630    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:53:09.877195 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:53:09.877137    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:53:31.354330 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:53:31.354282    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:53:31.355194 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:53:31.355133    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:53:39.877607 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:53:39.877474    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:54:09.877485 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:54:09.877446    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:54:39.877071 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:54:39.877032    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:55:09.877897 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:55:09.877819    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:55:39.876852 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:55:39.876817    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:55:46.355183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:55:46.351389    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:55:46.355183 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:55:46.351476    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:56:09.944442 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:56:09.944361    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:56:39.877455 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:56:39.877366    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:56:59.059594 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:56:59.059559    4531 generic.go:356] "Generic (PLEG): container finished" podID="61eb856b-5461-45bb-9126-c9d33d325ae7" containerID="b23399c1e3480f8a0fe451531a1b5001afd9a896c92e69ebffd55ba14b97889a" exitCode=1
Nov 03 01:56:59.060034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:56:59.059634    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb" event={"ID":"61eb856b-5461-45bb-9126-c9d33d325ae7","Type":"ContainerDied","Data":"b23399c1e3480f8a0fe451531a1b5001afd9a896c92e69ebffd55ba14b97889a"}
Nov 03 01:56:59.060034 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:56:59.059977    4531 scope.go:122] "RemoveContainer" containerID="b23399c1e3480f8a0fe451531a1b5001afd9a896c92e69ebffd55ba14b97889a"
Nov 03 01:56:59.060366 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:56:59.060351    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:57:00.068389 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:57:00.068304    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb" event={"ID":"61eb856b-5461-45bb-9126-c9d33d325ae7","Type":"ContainerStarted","Data":"7d8ebcfd1dd6324ce0c19b2c9bccc28e3ddc934cb4664219047dcf4e44f353e2"}
Nov 03 01:57:00.068890 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:57:00.068505    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 01:57:09.876733 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:57:09.876700    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:57:31.079970 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:57:31.079939    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mpgsb"
Nov 03 01:57:39.877538 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:57:39.877499    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:58:09.877148 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:58:09.877070    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:58:39.877170 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:58:39.877117    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:59:09.877878 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:59:09.877830    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:59:39.879284 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:59:39.879204    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 01:59:41.204478 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.204438    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 01:59:41.205004 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.204687    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" podUID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" containerName="manager" containerID="cri-o://88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946" gracePeriod=10
Nov 03 01:59:41.760086 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.760059    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 01:59:41.886855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.886766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\") pod \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") "
Nov 03 01:59:41.886855 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.886849    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\" (UniqueName: \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\") pod \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\" (UID: \"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd\") "
Nov 03 01:59:41.887155 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.887129    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume" pod "78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" (UID: "78d9bc3e-fee0-499f-9c05-f6fcdd169cfd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:41.888824 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.888798    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn" pod "78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" (UID: "78d9bc3e-fee0-499f-9c05-f6fcdd169cfd"). InnerVolumeSpecName "kube-api-access-6jlrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:41.988002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.987967    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlrn\" (UniqueName: \"kubernetes.io/projected/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-kube-api-access-6jlrn\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:41.988002 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:41.987996    4531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd-extensions-socket-volume\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:42.005863 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.005827    4531 generic.go:356] "Generic (PLEG): container finished" podID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" containerID="88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946" exitCode=0
Nov 03 01:59:42.005999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.005870    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"
Nov 03 01:59:42.005999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.005896    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" event={"ID":"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd","Type":"ContainerDied","Data":"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"}
Nov 03 01:59:42.005999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.005951    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9" event={"ID":"78d9bc3e-fee0-499f-9c05-f6fcdd169cfd","Type":"ContainerDied","Data":"775105cf7e2758b842c64c37a856296613f7af4812999e2ad0db03666f351af8"}
Nov 03 01:59:42.005999 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.005972    4531 scope.go:122] "RemoveContainer" containerID="88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"
Nov 03 01:59:42.023989 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.023964    4531 scope.go:122] "RemoveContainer" containerID="88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"
Nov 03 01:59:42.024338 ip-10-0-136-112 kubenswrapper[4531]: E1103 01:59:42.024318    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946\": container with ID starting with 88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946 not found: ID does not exist" containerID="88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"
Nov 03 01:59:42.024395 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.024349    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946"} err="failed to get container status \"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946\": rpc error: code = NotFound desc = could not find container \"88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946\": container with ID starting with 88c9c07aabbb09ea9e5dc7d742e368ee3f338d4dc679d68fc431290025314946 not found: ID does not exist"
Nov 03 01:59:42.034984 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.034959    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 01:59:42.038071 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.038049    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zsk9"]
Nov 03 01:59:42.901590 ip-10-0-136-112 kubenswrapper[4531]: I1103 01:59:42.901555    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" path="/var/lib/kubelet/pods/78d9bc3e-fee0-499f-9c05-f6fcdd169cfd/volumes"
Nov 03 02:00:09.876886 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:00:09.876847    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:00:39.876581 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:00:39.876538    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:00:42.226093 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226058    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"]
Nov 03 02:00:42.226545 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226531    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c0cda9d8-ab58-4714-a79a-70e7647474fc" containerName="maas-api"
Nov 03 02:00:42.226591 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226546    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cda9d8-ab58-4714-a79a-70e7647474fc" containerName="maas-api"
Nov 03 02:00:42.226591 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226559    4531 cpu_manager.go:414] "containerMap: removing container" podUID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" containerName="manager"
Nov 03 02:00:42.226591 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226565    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" containerName="manager"
Nov 03 02:00:42.226693 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226624    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cda9d8-ab58-4714-a79a-70e7647474fc" containerName="maas-api"
Nov 03 02:00:42.226693 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.226634    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d9bc3e-fee0-499f-9c05-f6fcdd169cfd" containerName="manager"
Nov 03 02:00:42.229830 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.229810    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.233918 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.233897    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2mtm8\""
Nov 03 02:00:42.241449 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.241421    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"]
Nov 03 02:00:42.361791 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.361758    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.361978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.361878    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzqq\" (UniqueName: \"kubernetes.io/projected/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-kube-api-access-7wzqq\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.462908 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.462864    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.463092 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.462964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzqq\" (UniqueName: \"kubernetes.io/projected/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-kube-api-access-7wzqq\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.463367 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.463346    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.471480 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.471456    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzqq\" (UniqueName: \"kubernetes.io/projected/20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e-kube-api-access-7wzqq\") pod \"kuadrant-operator-controller-manager-678d87766f-btqdp\" (UID: \"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.544535 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.544433    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:42.826957 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:42.826933    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"]
Nov 03 02:00:42.829773 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:00:42.829740    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d34f70_7eb9_44e6_a9e4_9ee6d6380d9e.slice/crio-6bdb9ca1da56a1b2f7ce5db7effb7c02a3bfbbbd1a56a30ce993f84a00f0950b WatchSource:0}: Error finding container 6bdb9ca1da56a1b2f7ce5db7effb7c02a3bfbbbd1a56a30ce993f84a00f0950b: Status 404 returned error can't find the container with id 6bdb9ca1da56a1b2f7ce5db7effb7c02a3bfbbbd1a56a30ce993f84a00f0950b
Nov 03 02:00:43.327570 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:43.327537    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp" event={"ID":"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e","Type":"ContainerStarted","Data":"673e188bc2712655ff6e9ed6c14dda2e0ade6c8f0137d225163afed73f1d88a5"}
Nov 03 02:00:43.327980 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:43.327587    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp" event={"ID":"20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e","Type":"ContainerStarted","Data":"6bdb9ca1da56a1b2f7ce5db7effb7c02a3bfbbbd1a56a30ce993f84a00f0950b"}
Nov 03 02:00:43.327980 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:43.327631    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:00:43.353234 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:43.353188    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp" podStartSLOduration=1.353173801 podStartE2EDuration="1.353173801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:43.351340244 +0000 UTC m=+4788.565083928" watchObservedRunningTime="2026-11-03 02:00:43.353173801 +0000 UTC m=+4788.566917507"
Nov 03 02:00:54.336308 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:00:54.336273    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-btqdp"
Nov 03 02:01:09.876703 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:01:09.876616    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:01:39.877934 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:01:39.877895    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:01:51.488431 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:51.488392    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 02:01:51.488869 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:51.488594    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" podUID="4f3c4620-dca5-46ac-8e75-8088927931c0" containerName="maas-api" containerID="cri-o://e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152" gracePeriod=30
Nov 03 02:01:52.729808 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.729784    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 02:01:52.791431 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.791355    4531 generic.go:356] "Generic (PLEG): container finished" podID="4f3c4620-dca5-46ac-8e75-8088927931c0" containerID="e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152" exitCode=0
Nov 03 02:01:52.791431 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.791402    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"
Nov 03 02:01:52.791617 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.791439    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" event={"ID":"4f3c4620-dca5-46ac-8e75-8088927931c0","Type":"ContainerDied","Data":"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"}
Nov 03 02:01:52.791617 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.791496    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r" event={"ID":"4f3c4620-dca5-46ac-8e75-8088927931c0","Type":"ContainerDied","Data":"b868999699f37194752f3163e274efe05d309945f65e80be25fa7fc45bbcd0a4"}
Nov 03 02:01:52.791617 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.791520    4531 scope.go:122] "RemoveContainer" containerID="e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"
Nov 03 02:01:52.801081 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.801060    4531 scope.go:122] "RemoveContainer" containerID="e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"
Nov 03 02:01:52.801462 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:01:52.801444    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152\": container with ID starting with e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152 not found: ID does not exist" containerID="e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"
Nov 03 02:01:52.801548 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.801471    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152"} err="failed to get container status \"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152\": rpc error: code = NotFound desc = could not find container \"e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152\": container with ID starting with e7a9414506adeb28bb27a16719dcc726651e1b7386f060436dd9c082990af152 not found: ID does not exist"
Nov 03 02:01:52.848141 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.848118    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\") pod \"4f3c4620-dca5-46ac-8e75-8088927931c0\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") "
Nov 03 02:01:52.848262 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.848185    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\" (UniqueName: \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\") pod \"4f3c4620-dca5-46ac-8e75-8088927931c0\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") "
Nov 03 02:01:52.848262 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.848231    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\") pod \"4f3c4620-dca5-46ac-8e75-8088927931c0\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") "
Nov 03 02:01:52.848373 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.848356    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\") pod \"4f3c4620-dca5-46ac-8e75-8088927931c0\" (UID: \"4f3c4620-dca5-46ac-8e75-8088927931c0\") "
Nov 03 02:01:52.848727 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.848701    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca" pod "4f3c4620-dca5-46ac-8e75-8088927931c0" (UID: "4f3c4620-dca5-46ac-8e75-8088927931c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:52.850200 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.850175    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs" pod "4f3c4620-dca5-46ac-8e75-8088927931c0" (UID: "4f3c4620-dca5-46ac-8e75-8088927931c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:52.850292 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.850196    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls" pod "4f3c4620-dca5-46ac-8e75-8088927931c0" (UID: "4f3c4620-dca5-46ac-8e75-8088927931c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:52.850292 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.850236    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68" pod "4f3c4620-dca5-46ac-8e75-8088927931c0" (UID: "4f3c4620-dca5-46ac-8e75-8088927931c0"). InnerVolumeSpecName "kube-api-access-p2t68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:52.949113 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.949078    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3c4620-dca5-46ac-8e75-8088927931c0-openshift-service-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:52.949113 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.949109    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-metrics-certs\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:52.949319 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.949124    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2t68\" (UniqueName: \"kubernetes.io/projected/4f3c4620-dca5-46ac-8e75-8088927931c0-kube-api-access-p2t68\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:52.949319 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:52.949138    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f3c4620-dca5-46ac-8e75-8088927931c0-maas-api-tls\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:53.109091 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:53.109060    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 02:01:53.113649 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:53.113623    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cc657fcb-sst8r"]
Nov 03 02:01:54.902389 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:01:54.902355    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3c4620-dca5-46ac-8e75-8088927931c0" path="/var/lib/kubelet/pods/4f3c4620-dca5-46ac-8e75-8088927931c0/volumes"
Nov 03 02:02:01.402502 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.402461    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"]
Nov 03 02:02:01.402962 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.402946    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4f3c4620-dca5-46ac-8e75-8088927931c0" containerName="maas-api"
Nov 03 02:02:01.403015 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.402963    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3c4620-dca5-46ac-8e75-8088927931c0" containerName="maas-api"
Nov 03 02:02:01.403052 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.403044    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3c4620-dca5-46ac-8e75-8088927931c0" containerName="maas-api"
Nov 03 02:02:01.407621 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.407593    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.411380 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.411323    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:01.411674 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.411649    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:02:01.413909 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.413884    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"]
Nov 03 02:02:01.530911 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.530876    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.531098 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.530917    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lc2\" (UniqueName: \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.531098 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.531037    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.632569 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.632522    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lc2\" (UniqueName: \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.632772 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.632597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.632772 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.632686    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.633353 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.633331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.635260 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.635232    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.641373 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.641346    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lc2\" (UniqueName: \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\") pod \"maas-api-revoke-sub-e8505b62-bgsxn\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:01.724231 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:01.724116    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:02.000106 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:02.000074    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"]
Nov 03 02:02:02.002944 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:02:02.002911    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf522cd0e_7ac6_4271_a30c_5f1f825c9ea3.slice/crio-7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205 WatchSource:0}: Error finding container 7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205: Status 404 returned error can't find the container with id 7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205
Nov 03 02:02:02.848729 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:02.848581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn" event={"ID":"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3","Type":"ContainerStarted","Data":"f888b097929b9c06cdfaa1574e020ebe7f1d7dbd1bd73c7ab78b8be81e51be4e"}
Nov 03 02:02:02.848729 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:02.848660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn" event={"ID":"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3","Type":"ContainerStarted","Data":"7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205"}
Nov 03 02:02:03.859760 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:03.859726    4531 generic.go:356] "Generic (PLEG): container finished" podID="f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" containerID="f888b097929b9c06cdfaa1574e020ebe7f1d7dbd1bd73c7ab78b8be81e51be4e" exitCode=0
Nov 03 02:02:03.860224 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:03.859810    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn" event={"ID":"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3","Type":"ContainerDied","Data":"f888b097929b9c06cdfaa1574e020ebe7f1d7dbd1bd73c7ab78b8be81e51be4e"}
Nov 03 02:02:03.990629 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:03.990600    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:04.167487 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.167400    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\") pod \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") "
Nov 03 02:02:04.167487 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.167443    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\" (UniqueName: \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\") pod \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") "
Nov 03 02:02:04.167487 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.167474    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\") pod \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\" (UID: \"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3\") "
Nov 03 02:02:04.167848 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.167825    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca" pod "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" (UID: "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:04.169588 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.169564    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca" pod "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" (UID: "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:04.169740 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.169720    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2" pod "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" (UID: "f522cd0e-7ac6-4271-a30c-5f1f825c9ea3"). InnerVolumeSpecName "kube-api-access-f5lc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:04.268642 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.268608    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.268642 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.268635    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lc2\" (UniqueName: \"kubernetes.io/projected/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-kube-api-access-f5lc2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.268642 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.268646    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.867631 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.867601    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"
Nov 03 02:02:04.867631 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.867618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn" event={"ID":"f522cd0e-7ac6-4271-a30c-5f1f825c9ea3","Type":"ContainerDied","Data":"7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205"}
Nov 03 02:02:04.868079 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:04.867667    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7793dfacabe0d3951f995b95aba387e221db6332230395b6149f15f2db073205"
Nov 03 02:02:09.876904 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:02:09.876867    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:02:22.497299 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.497263    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"]
Nov 03 02:02:22.497924 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.497715    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" containerName="revoke-keys"
Nov 03 02:02:22.497924 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.497732    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" containerName="revoke-keys"
Nov 03 02:02:22.497924 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.497823    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" containerName="revoke-keys"
Nov 03 02:02:22.501052 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.501032    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.506560 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.506522    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:22.506741 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.506644    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:02:22.510006 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.509985    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"]
Nov 03 02:02:22.559073 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.559043    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.559270 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.559244    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.559340 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.559271    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.659938 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.659900    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.660118 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.659989    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.660118 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.660009    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.660650 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.660630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.662509 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.662488    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.670596 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.670573    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\") pod \"maas-api-revoke-sub-f979a8c5-kr4cp\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:22.817496 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:22.817412    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:23.092450 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:23.092421    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"]
Nov 03 02:02:23.093851 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:02:23.093822    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594e95cc_d9c4_4fe4_ad0f_334dbdc193f8.slice/crio-ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33 WatchSource:0}: Error finding container ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33: Status 404 returned error can't find the container with id ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33
Nov 03 02:02:23.998245 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:23.998193    4531 generic.go:356] "Generic (PLEG): container finished" podID="594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" containerID="8465a56c9e2238caf9517519f01fe9afb85fb861071cdeb63169179762d5458a" exitCode=0
Nov 03 02:02:23.998634 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:23.998282    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp" event={"ID":"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8","Type":"ContainerDied","Data":"8465a56c9e2238caf9517519f01fe9afb85fb861071cdeb63169179762d5458a"}
Nov 03 02:02:23.998634 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:23.998331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp" event={"ID":"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8","Type":"ContainerStarted","Data":"ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33"}
Nov 03 02:02:25.137182 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.137143    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:25.189185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.185684    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\") pod \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") "
Nov 03 02:02:25.189185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.185827    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\") pod \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") "
Nov 03 02:02:25.189185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.185865    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\") pod \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\" (UID: \"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8\") "
Nov 03 02:02:25.189185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.186785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca" pod "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" (UID: "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:25.194076 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.194042    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca" pod "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" (UID: "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:25.194256 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.194084    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n" pod "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" (UID: "594e95cc-d9c4-4fe4-ad0f-334dbdc193f8"). InnerVolumeSpecName "kube-api-access-8m56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:25.287254 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.287145    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-kube-api-access-8m56n\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:25.287254 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.287203    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:25.287254 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:25.287220    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594e95cc-d9c4-4fe4-ad0f-334dbdc193f8-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.011072 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:26.011043    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"
Nov 03 02:02:26.011072 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:26.011059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp" event={"ID":"594e95cc-d9c4-4fe4-ad0f-334dbdc193f8","Type":"ContainerDied","Data":"ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33"}
Nov 03 02:02:26.011333 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:26.011101    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3e716a9b2a56ee65cfca22d9d7e05cdd6dd8858937859fa6771b8260b08e33"
Nov 03 02:02:39.994239 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:02:39.993531    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:02:48.725682 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.725642    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"]
Nov 03 02:02:48.726274 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.726243    4531 cpu_manager.go:414] "containerMap: removing container" podUID="594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" containerName="revoke-keys"
Nov 03 02:02:48.726274 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.726266    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" containerName="revoke-keys"
Nov 03 02:02:48.726459 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.726372    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="594e95cc-d9c4-4fe4-ad0f-334dbdc193f8" containerName="revoke-keys"
Nov 03 02:02:48.729623 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.729607    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.734573 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.734246    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:48.734799 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.734778    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:02:48.735894 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.735873    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"]
Nov 03 02:02:48.784506 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.784462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.784706 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.784590    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhwv\" (UniqueName: \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.784706 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.784627    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.885839 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.885785    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.886039 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.885873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.886039 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.885938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhwv\" (UniqueName: \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.886590 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.886563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.888243 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.888199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:48.893820 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:48.893797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhwv\" (UniqueName: \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\") pod \"maas-api-revoke-sub-c0af0409-h9wrb\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:49.045986 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:49.045899    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:49.320619 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:49.320593    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"]
Nov 03 02:02:49.321630 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:02:49.321604    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea15f1c_1d1b_4299_bf6a_530425ace9ce.slice/crio-817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4 WatchSource:0}: Error finding container 817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4: Status 404 returned error can't find the container with id 817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4
Nov 03 02:02:50.256298 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:50.256269    4531 generic.go:356] "Generic (PLEG): container finished" podID="9ea15f1c-1d1b-4299-bf6a-530425ace9ce" containerID="48c81323c6dca0963a90b8a1e540fd867c3d26bf024062adea652eb07c25d915" exitCode=0
Nov 03 02:02:50.256863 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:50.256353    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb" event={"ID":"9ea15f1c-1d1b-4299-bf6a-530425ace9ce","Type":"ContainerDied","Data":"48c81323c6dca0963a90b8a1e540fd867c3d26bf024062adea652eb07c25d915"}
Nov 03 02:02:50.256863 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:50.256390    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb" event={"ID":"9ea15f1c-1d1b-4299-bf6a-530425ace9ce","Type":"ContainerStarted","Data":"817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4"}
Nov 03 02:02:51.398564 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.398538    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:51.510293 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.510255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\") pod \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") "
Nov 03 02:02:51.510498 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.510408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\" (UniqueName: \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\") pod \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") "
Nov 03 02:02:51.510498 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.510458    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\") pod \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\" (UID: \"9ea15f1c-1d1b-4299-bf6a-530425ace9ce\") "
Nov 03 02:02:51.510836 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.510810    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca" pod "9ea15f1c-1d1b-4299-bf6a-530425ace9ce" (UID: "9ea15f1c-1d1b-4299-bf6a-530425ace9ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:51.512489 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.512435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv" pod "9ea15f1c-1d1b-4299-bf6a-530425ace9ce" (UID: "9ea15f1c-1d1b-4299-bf6a-530425ace9ce"). InnerVolumeSpecName "kube-api-access-djhwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:51.512489 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.512449    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca" pod "9ea15f1c-1d1b-4299-bf6a-530425ace9ce" (UID: "9ea15f1c-1d1b-4299-bf6a-530425ace9ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:51.611387 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.611356    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:51.611387 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.611381    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhwv\" (UniqueName: \"kubernetes.io/projected/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-kube-api-access-djhwv\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:51.611387 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:51.611393    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ea15f1c-1d1b-4299-bf6a-530425ace9ce-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:52.269903 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:52.269795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb" event={"ID":"9ea15f1c-1d1b-4299-bf6a-530425ace9ce","Type":"ContainerDied","Data":"817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4"}
Nov 03 02:02:52.269903 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:52.269873    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817301e06f0841f528f23bea80649800d940f04782ccc2677b8bb44a2d85c9f4"
Nov 03 02:02:52.270386 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:52.270324    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0af0409-h9wrb"
Nov 03 02:02:59.061341 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.061283    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"]
Nov 03 02:02:59.061977 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.061956    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9ea15f1c-1d1b-4299-bf6a-530425ace9ce" containerName="revoke-keys"
Nov 03 02:02:59.062078 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.061978    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea15f1c-1d1b-4299-bf6a-530425ace9ce" containerName="revoke-keys"
Nov 03 02:02:59.062188 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.062155    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea15f1c-1d1b-4299-bf6a-530425ace9ce" containerName="revoke-keys"
Nov 03 02:02:59.066029 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.066009    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.070032 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.069828    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:59.071619 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.071563    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:02:59.084155 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.072381    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"]
Nov 03 02:02:59.184218 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.184147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.184382 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.184356    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkpp\" (UniqueName: \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.184462 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.184407    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.285949 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.285907    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.286196 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.285990    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.286196 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.286103    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkpp\" (UniqueName: \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.286816 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.286790    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.288348 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.288325    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.294517 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.294496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkpp\" (UniqueName: \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\") pod \"maas-api-revoke-sub-81e14ee0-hknxh\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.397886 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.397849    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:02:59.676594 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:02:59.676567    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"]
Nov 03 02:02:59.679327 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:02:59.679297    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6e09da_340a_4189_ba4c_4b806232cc65.slice/crio-22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5 WatchSource:0}: Error finding container 22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5: Status 404 returned error can't find the container with id 22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5
Nov 03 02:03:00.317074 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:00.316984    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh" event={"ID":"3b6e09da-340a-4189-ba4c-4b806232cc65","Type":"ContainerStarted","Data":"495eeb6e88835c3babaa2763673f034a8f1672ab067e37c63fd00bdceafa52d0"}
Nov 03 02:03:00.317074 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:00.317042    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh" event={"ID":"3b6e09da-340a-4189-ba4c-4b806232cc65","Type":"ContainerStarted","Data":"22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5"}
Nov 03 02:03:00.335870 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:00.335822    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh" podStartSLOduration=1.335806451 podStartE2EDuration="1.335806451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:00.33233222 +0000 UTC m=+4925.546075904" watchObservedRunningTime="2026-11-03 02:03:00.335806451 +0000 UTC m=+4925.549550121"
Nov 03 02:03:01.324352 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:01.324308    4531 generic.go:356] "Generic (PLEG): container finished" podID="3b6e09da-340a-4189-ba4c-4b806232cc65" containerID="495eeb6e88835c3babaa2763673f034a8f1672ab067e37c63fd00bdceafa52d0" exitCode=0
Nov 03 02:03:01.324723 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:01.324368    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh" event={"ID":"3b6e09da-340a-4189-ba4c-4b806232cc65","Type":"ContainerDied","Data":"495eeb6e88835c3babaa2763673f034a8f1672ab067e37c63fd00bdceafa52d0"}
Nov 03 02:03:02.460507 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.460478    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:03:02.519257 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.519224    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\") pod \"3b6e09da-340a-4189-ba4c-4b806232cc65\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") "
Nov 03 02:03:02.519427 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.519342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\" (UniqueName: \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\") pod \"3b6e09da-340a-4189-ba4c-4b806232cc65\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") "
Nov 03 02:03:02.519427 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.519372    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\") pod \"3b6e09da-340a-4189-ba4c-4b806232cc65\" (UID: \"3b6e09da-340a-4189-ba4c-4b806232cc65\") "
Nov 03 02:03:02.519620 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.519594    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca" pod "3b6e09da-340a-4189-ba4c-4b806232cc65" (UID: "3b6e09da-340a-4189-ba4c-4b806232cc65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:02.521530 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.521505    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca" pod "3b6e09da-340a-4189-ba4c-4b806232cc65" (UID: "3b6e09da-340a-4189-ba4c-4b806232cc65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:02.521663 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.521568    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp" pod "3b6e09da-340a-4189-ba4c-4b806232cc65" (UID: "3b6e09da-340a-4189-ba4c-4b806232cc65"). InnerVolumeSpecName "kube-api-access-pzkpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:02.620291 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.620211    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzkpp\" (UniqueName: \"kubernetes.io/projected/3b6e09da-340a-4189-ba4c-4b806232cc65-kube-api-access-pzkpp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:02.620291 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.620239    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:02.620291 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:02.620251    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e09da-340a-4189-ba4c-4b806232cc65-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:03.336392 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:03.336361    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh"
Nov 03 02:03:03.336601 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:03.336399    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e14ee0-hknxh" event={"ID":"3b6e09da-340a-4189-ba4c-4b806232cc65","Type":"ContainerDied","Data":"22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5"}
Nov 03 02:03:03.336601 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:03.336466    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22cce608daceb913989c22634dee57747bbcf8fe93373ffa9d8b2af954dae1a5"
Nov 03 02:03:09.877621 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:03:09.877571    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:03:25.560631 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.560588    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"]
Nov 03 02:03:25.561286 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.561265    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3b6e09da-340a-4189-ba4c-4b806232cc65" containerName="revoke-keys"
Nov 03 02:03:25.561393 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.561287    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6e09da-340a-4189-ba4c-4b806232cc65" containerName="revoke-keys"
Nov 03 02:03:25.561452 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.561437    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6e09da-340a-4189-ba4c-4b806232cc65" containerName="revoke-keys"
Nov 03 02:03:25.564706 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.564684    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.569804 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.569780    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:25.569957 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.569878    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:03:25.573192 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.573143    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"]
Nov 03 02:03:25.622818 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.622780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.622992 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.622893    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.623039 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.622985    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwm2\" (UniqueName: \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.723877 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.723844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.724063 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.723887    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwm2\" (UniqueName: \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.724063 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.723991    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.724670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.724640    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.726172 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.726136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.733242 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.733220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwm2\" (UniqueName: \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\") pod \"maas-api-revoke-sub-25c1e8b5-lgpxl\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:25.878842 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:25.878809    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:26.158323 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:26.158299    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"]
Nov 03 02:03:26.159903 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:03:26.159880    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1323b7_f732_45f6_80af_be127525fbcc.slice/crio-df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9 WatchSource:0}: Error finding container df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9: Status 404 returned error can't find the container with id df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9
Nov 03 02:03:26.523252 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:26.523205    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl" event={"ID":"ac1323b7-f732-45f6-80af-be127525fbcc","Type":"ContainerStarted","Data":"5419344848e92d7ffc8b62774aca5184c05475891fbcc30d14cf35de274abf81"}
Nov 03 02:03:26.523427 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:26.523268    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl" event={"ID":"ac1323b7-f732-45f6-80af-be127525fbcc","Type":"ContainerStarted","Data":"df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9"}
Nov 03 02:03:26.540895 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:26.540846    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl" podStartSLOduration=1.54083293 podStartE2EDuration="1.54083293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:26.538323725 +0000 UTC m=+4951.752067430" watchObservedRunningTime="2026-11-03 02:03:26.54083293 +0000 UTC m=+4951.754576601"
Nov 03 02:03:27.530818 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:27.530779    4531 generic.go:356] "Generic (PLEG): container finished" podID="ac1323b7-f732-45f6-80af-be127525fbcc" containerID="5419344848e92d7ffc8b62774aca5184c05475891fbcc30d14cf35de274abf81" exitCode=0
Nov 03 02:03:27.531240 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:27.530865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl" event={"ID":"ac1323b7-f732-45f6-80af-be127525fbcc","Type":"ContainerDied","Data":"5419344848e92d7ffc8b62774aca5184c05475891fbcc30d14cf35de274abf81"}
Nov 03 02:03:28.662925 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.662899    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:28.749200 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.749150    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\") pod \"ac1323b7-f732-45f6-80af-be127525fbcc\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") "
Nov 03 02:03:28.749387 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.749224    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\" (UniqueName: \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\") pod \"ac1323b7-f732-45f6-80af-be127525fbcc\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") "
Nov 03 02:03:28.749387 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.749316    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\") pod \"ac1323b7-f732-45f6-80af-be127525fbcc\" (UID: \"ac1323b7-f732-45f6-80af-be127525fbcc\") "
Nov 03 02:03:28.749577 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.749549    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca" pod "ac1323b7-f732-45f6-80af-be127525fbcc" (UID: "ac1323b7-f732-45f6-80af-be127525fbcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:28.751216 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.751186    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca" pod "ac1323b7-f732-45f6-80af-be127525fbcc" (UID: "ac1323b7-f732-45f6-80af-be127525fbcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:28.751330 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.751304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2" pod "ac1323b7-f732-45f6-80af-be127525fbcc" (UID: "ac1323b7-f732-45f6-80af-be127525fbcc"). InnerVolumeSpecName "kube-api-access-zpwm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:28.850104 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.850070    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:28.850104 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.850098    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwm2\" (UniqueName: \"kubernetes.io/projected/ac1323b7-f732-45f6-80af-be127525fbcc-kube-api-access-zpwm2\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:28.850104 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:28.850108    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1323b7-f732-45f6-80af-be127525fbcc-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:29.543865 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:29.543830    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl"
Nov 03 02:03:29.544089 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:29.543873    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c1e8b5-lgpxl" event={"ID":"ac1323b7-f732-45f6-80af-be127525fbcc","Type":"ContainerDied","Data":"df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9"}
Nov 03 02:03:29.544089 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:29.543920    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6183529b0aa8e07069665ed2443c898ea06ac3f1544cdcf8eb5b829f4519f9"
Nov 03 02:03:35.762532 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.762490    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"]
Nov 03 02:03:35.762989 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.762973    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ac1323b7-f732-45f6-80af-be127525fbcc" containerName="revoke-keys"
Nov 03 02:03:35.763034 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.762991    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1323b7-f732-45f6-80af-be127525fbcc" containerName="revoke-keys"
Nov 03 02:03:35.763077 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.763064    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1323b7-f732-45f6-80af-be127525fbcc" containerName="revoke-keys"
Nov 03 02:03:35.766444 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.766426    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.770408 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.770379    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:35.770560 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.770461    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:03:35.773236 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.773196    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"]
Nov 03 02:03:35.819466 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.819426    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvznk\" (UniqueName: \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.819670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.819601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.819731 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.819709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.920305 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.920256    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.920514 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.920358    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvznk\" (UniqueName: \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.920514 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.920452    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.920997 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.920970    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.922734 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.922715    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:35.928346 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:35.928324    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvznk\" (UniqueName: \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\") pod \"maas-api-revoke-sub-4bce2bf8-z92cm\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:36.082746 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:36.082718    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:36.360531 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:36.360449    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"]
Nov 03 02:03:36.363882 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:03:36.363841    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a54bea8_6445_4634_9c1b_091a34d15c06.slice/crio-25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87 WatchSource:0}: Error finding container 25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87: Status 404 returned error can't find the container with id 25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87
Nov 03 02:03:36.583954 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:36.583920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm" event={"ID":"8a54bea8-6445-4634-9c1b-091a34d15c06","Type":"ContainerStarted","Data":"25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87"}
Nov 03 02:03:37.607803 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:37.607750    4531 generic.go:356] "Generic (PLEG): container finished" podID="8a54bea8-6445-4634-9c1b-091a34d15c06" containerID="671f9295db0df798df365ff5a232d2cb349215890f7c80c6bb8edc7aa8f4e863" exitCode=0
Nov 03 02:03:37.608369 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:37.608006    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm" event={"ID":"8a54bea8-6445-4634-9c1b-091a34d15c06","Type":"ContainerDied","Data":"671f9295db0df798df365ff5a232d2cb349215890f7c80c6bb8edc7aa8f4e863"}
Nov 03 02:03:38.804185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:38.803296    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-d852p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:03:38.804185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:38.803366    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-d852p" podUID="c9342480-831c-4417-8711-a69e79b817a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:03:39.085188 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.081352    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:39.124218 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.124155    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\" (UniqueName: \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\") pod \"8a54bea8-6445-4634-9c1b-091a34d15c06\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") "
Nov 03 02:03:39.124410 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.124259    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\") pod \"8a54bea8-6445-4634-9c1b-091a34d15c06\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") "
Nov 03 02:03:39.126263 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.126233    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca" pod "8a54bea8-6445-4634-9c1b-091a34d15c06" (UID: "8a54bea8-6445-4634-9c1b-091a34d15c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:39.126448 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.126362    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk" pod "8a54bea8-6445-4634-9c1b-091a34d15c06" (UID: "8a54bea8-6445-4634-9c1b-091a34d15c06"). InnerVolumeSpecName "kube-api-access-tvznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:39.225183 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.224903    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\") pod \"8a54bea8-6445-4634-9c1b-091a34d15c06\" (UID: \"8a54bea8-6445-4634-9c1b-091a34d15c06\") "
Nov 03 02:03:39.225369 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.225257    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:39.225369 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.225280    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvznk\" (UniqueName: \"kubernetes.io/projected/8a54bea8-6445-4634-9c1b-091a34d15c06-kube-api-access-tvznk\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:39.225369 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.225290    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca" pod "8a54bea8-6445-4634-9c1b-091a34d15c06" (UID: "8a54bea8-6445-4634-9c1b-091a34d15c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:39.326486 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.326448    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a54bea8-6445-4634-9c1b-091a34d15c06-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:39.624290 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.624210    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm"
Nov 03 02:03:39.624290 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.624236    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bce2bf8-z92cm" event={"ID":"8a54bea8-6445-4634-9c1b-091a34d15c06","Type":"ContainerDied","Data":"25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87"}
Nov 03 02:03:39.624290 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:03:39.624284    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25542ebe9da23e527a59eb95c7d9358b1ce6cc79e08d25adcdfffd1cf8a5bf87"
Nov 03 02:03:39.878636 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:03:39.878541    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:04:01.354183 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:01.353273    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-596dc9545f-46hnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:04:01.354183 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:01.353347    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-596dc9545f-46hnd" podUID="37b1042d-911e-4e83-b83d-dbb8096646d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:04:02.911061 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.911026    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"]
Nov 03 02:04:02.911502 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.911487    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8a54bea8-6445-4634-9c1b-091a34d15c06" containerName="revoke-keys"
Nov 03 02:04:02.911549 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.911504    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a54bea8-6445-4634-9c1b-091a34d15c06" containerName="revoke-keys"
Nov 03 02:04:02.911602 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.911592    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a54bea8-6445-4634-9c1b-091a34d15c06" containerName="revoke-keys"
Nov 03 02:04:02.914764 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.914746    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:02.919312 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.919286    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:02.919449 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.919417    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:02.923526 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:02.923500    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"]
Nov 03 02:04:03.035531 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.035493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.035709 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.035554    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwc55\" (UniqueName: \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.035709 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.035667    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.136848 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.136796    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.137036 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.136873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwc55\" (UniqueName: \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.137036 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.136910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.137640 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.137613    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.139358 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.139335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.145649 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.145628    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwc55\" (UniqueName: \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\") pod \"maas-api-revoke-sub-8d440fa1-l2lrp\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.231433 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.231352    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:03.508421 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.508395    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"]
Nov 03 02:04:03.510053 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:03.510027    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bddff25_15ae_43bb_8846_097b492619a9.slice/crio-29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be WatchSource:0}: Error finding container 29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be: Status 404 returned error can't find the container with id 29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be
Nov 03 02:04:03.831624 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:03.831595    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp" event={"ID":"4bddff25-15ae-43bb-8846-097b492619a9","Type":"ContainerStarted","Data":"29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be"}
Nov 03 02:04:04.839019 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:04.838988    4531 generic.go:356] "Generic (PLEG): container finished" podID="4bddff25-15ae-43bb-8846-097b492619a9" containerID="3b880321be140bc16c625af38292ba14ac33db92b6d1ce59299bb9aa50959f63" exitCode=0
Nov 03 02:04:04.839428 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:04.839080    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp" event={"ID":"4bddff25-15ae-43bb-8846-097b492619a9","Type":"ContainerDied","Data":"3b880321be140bc16c625af38292ba14ac33db92b6d1ce59299bb9aa50959f63"}
Nov 03 02:04:05.982859 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:05.982832    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:06.065459 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.065427    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\") pod \"4bddff25-15ae-43bb-8846-097b492619a9\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") "
Nov 03 02:04:06.065637 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.065579    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\") pod \"4bddff25-15ae-43bb-8846-097b492619a9\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") "
Nov 03 02:04:06.065637 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.065600    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\" (UniqueName: \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\") pod \"4bddff25-15ae-43bb-8846-097b492619a9\" (UID: \"4bddff25-15ae-43bb-8846-097b492619a9\") "
Nov 03 02:04:06.065978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.065941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca" pod "4bddff25-15ae-43bb-8846-097b492619a9" (UID: "4bddff25-15ae-43bb-8846-097b492619a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:06.067533 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.067508    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca" pod "4bddff25-15ae-43bb-8846-097b492619a9" (UID: "4bddff25-15ae-43bb-8846-097b492619a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:06.067644 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.067616    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55" pod "4bddff25-15ae-43bb-8846-097b492619a9" (UID: "4bddff25-15ae-43bb-8846-097b492619a9"). InnerVolumeSpecName "kube-api-access-lwc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:06.166951 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.166870    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:06.166951 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.166900    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwc55\" (UniqueName: \"kubernetes.io/projected/4bddff25-15ae-43bb-8846-097b492619a9-kube-api-access-lwc55\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:06.166951 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.166909    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bddff25-15ae-43bb-8846-097b492619a9-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:06.851591 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.851508    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp"
Nov 03 02:04:06.851591 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.851548    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d440fa1-l2lrp" event={"ID":"4bddff25-15ae-43bb-8846-097b492619a9","Type":"ContainerDied","Data":"29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be"}
Nov 03 02:04:06.851842 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:06.851627    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29465c315177d940dab204ecb4574eae0d8b6318ef4dfb18019aad77ed6874be"
Nov 03 02:04:09.877458 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:04:09.877422    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:04:13.133314 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.133280    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"]
Nov 03 02:04:13.133737 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.133724    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4bddff25-15ae-43bb-8846-097b492619a9" containerName="revoke-keys"
Nov 03 02:04:13.133795 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.133738    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bddff25-15ae-43bb-8846-097b492619a9" containerName="revoke-keys"
Nov 03 02:04:13.133842 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.133832    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bddff25-15ae-43bb-8846-097b492619a9" containerName="revoke-keys"
Nov 03 02:04:13.137074 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.137050    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.142522 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.142493    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:13.142841 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.142812    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:13.146949 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.146740    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"]
Nov 03 02:04:13.236695 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.236646    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.236905 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.236783    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.236905 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.236837    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f79r\" (UniqueName: \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.338045 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.338003    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.338273 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.338096    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.338273 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.338143    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f79r\" (UniqueName: \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.338840 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.338812    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.340631 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.340605    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.347228 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.347205    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f79r\" (UniqueName: \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\") pod \"maas-api-revoke-sub-33397942-w24lb\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.455732 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.455648    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:13.731874 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.731841    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"]
Nov 03 02:04:13.733779 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:13.733748    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc840156d_b881_4e59_9d76_e881a02147f6.slice/crio-7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f WatchSource:0}: Error finding container 7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f: Status 404 returned error can't find the container with id 7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f
Nov 03 02:04:13.892807 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:13.892765    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb" event={"ID":"c840156d-b881-4e59-9d76-e881a02147f6","Type":"ContainerStarted","Data":"7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f"}
Nov 03 02:04:14.906733 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:14.906691    4531 generic.go:356] "Generic (PLEG): container finished" podID="c840156d-b881-4e59-9d76-e881a02147f6" containerID="1eeecb75faa8efd5acab23cb1f457196d5724aa97230d234f1b7e226e387428d" exitCode=0
Nov 03 02:04:14.907263 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:14.906872    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb" event={"ID":"c840156d-b881-4e59-9d76-e881a02147f6","Type":"ContainerDied","Data":"1eeecb75faa8efd5acab23cb1f457196d5724aa97230d234f1b7e226e387428d"}
Nov 03 02:04:16.525772 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.525699    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:16.622820 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.622772    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\") pod \"c840156d-b881-4e59-9d76-e881a02147f6\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") "
Nov 03 02:04:16.623628 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.623484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca" pod "c840156d-b881-4e59-9d76-e881a02147f6" (UID: "c840156d-b881-4e59-9d76-e881a02147f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:16.623799 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.623619    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\") pod \"c840156d-b881-4e59-9d76-e881a02147f6\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") "
Nov 03 02:04:16.624024 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.624006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\" (UniqueName: \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\") pod \"c840156d-b881-4e59-9d76-e881a02147f6\" (UID: \"c840156d-b881-4e59-9d76-e881a02147f6\") "
Nov 03 02:04:16.624559 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.624527    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:16.634351 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.634296    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r" pod "c840156d-b881-4e59-9d76-e881a02147f6" (UID: "c840156d-b881-4e59-9d76-e881a02147f6"). InnerVolumeSpecName "kube-api-access-9f79r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:16.668183 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.654287    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca" pod "c840156d-b881-4e59-9d76-e881a02147f6" (UID: "c840156d-b881-4e59-9d76-e881a02147f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:16.725838 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.725796    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c840156d-b881-4e59-9d76-e881a02147f6-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:16.725838 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.725830    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f79r\" (UniqueName: \"kubernetes.io/projected/c840156d-b881-4e59-9d76-e881a02147f6-kube-api-access-9f79r\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:16.925124 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.925099    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb"
Nov 03 02:04:16.925359 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.925134    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33397942-w24lb" event={"ID":"c840156d-b881-4e59-9d76-e881a02147f6","Type":"ContainerDied","Data":"7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f"}
Nov 03 02:04:16.925359 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:16.925203    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf4991531080571d823234168bfd5ecffe2cefc81b5d52466dcd88f2ef3758f"
Nov 03 02:04:23.344941 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.344907    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"]
Nov 03 02:04:23.345411 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.345397    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c840156d-b881-4e59-9d76-e881a02147f6" containerName="revoke-keys"
Nov 03 02:04:23.345457 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.345412    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c840156d-b881-4e59-9d76-e881a02147f6" containerName="revoke-keys"
Nov 03 02:04:23.345516 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.345506    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c840156d-b881-4e59-9d76-e881a02147f6" containerName="revoke-keys"
Nov 03 02:04:23.350208 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.350105    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.356794 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.356758    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:23.357061 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.356805    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:23.360885 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.360858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"]
Nov 03 02:04:23.407440 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.407404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7cl\" (UniqueName: \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.407672 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.407460    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.407672 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.407568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.508563 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.508516    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.508758 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.508577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.508758 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.508645    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7cl\" (UniqueName: \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.509286 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.509259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.510964 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.510941    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.518978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.518955    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7cl\" (UniqueName: \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\") pod \"maas-api-revoke-sub-74560ab4-6rtt8\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.673110 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.673017    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:23.949037 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.949006    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"]
Nov 03 02:04:23.950791 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:23.950763    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012f6859_784b_4379_9979_6ed61173ce66.slice/crio-135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3 WatchSource:0}: Error finding container 135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3: Status 404 returned error can't find the container with id 135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3
Nov 03 02:04:23.970362 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:23.970327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8" event={"ID":"012f6859-784b-4379-9979-6ed61173ce66","Type":"ContainerStarted","Data":"135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3"}
Nov 03 02:04:24.978102 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:24.978062    4531 generic.go:356] "Generic (PLEG): container finished" podID="012f6859-784b-4379-9979-6ed61173ce66" containerID="12fbef8c71581d623e772c2f54508049759147d9f7d4aaf1749c9b4b38ca0a6c" exitCode=0
Nov 03 02:04:24.978590 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:24.978152    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8" event={"ID":"012f6859-784b-4379-9979-6ed61173ce66","Type":"ContainerDied","Data":"12fbef8c71581d623e772c2f54508049759147d9f7d4aaf1749c9b4b38ca0a6c"}
Nov 03 02:04:26.117015 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.116986    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:26.237396 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.237311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\" (UniqueName: \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\") pod \"012f6859-784b-4379-9979-6ed61173ce66\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") "
Nov 03 02:04:26.237396 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.237375    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\") pod \"012f6859-784b-4379-9979-6ed61173ce66\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") "
Nov 03 02:04:26.237598 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.237408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\") pod \"012f6859-784b-4379-9979-6ed61173ce66\" (UID: \"012f6859-784b-4379-9979-6ed61173ce66\") "
Nov 03 02:04:26.237840 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.237813    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca" pod "012f6859-784b-4379-9979-6ed61173ce66" (UID: "012f6859-784b-4379-9979-6ed61173ce66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:26.239496 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.239467    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl" pod "012f6859-784b-4379-9979-6ed61173ce66" (UID: "012f6859-784b-4379-9979-6ed61173ce66"). InnerVolumeSpecName "kube-api-access-df7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:26.239620 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.239593    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca" pod "012f6859-784b-4379-9979-6ed61173ce66" (UID: "012f6859-784b-4379-9979-6ed61173ce66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:26.338697 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.338664    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7cl\" (UniqueName: \"kubernetes.io/projected/012f6859-784b-4379-9979-6ed61173ce66-kube-api-access-df7cl\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:26.338697 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.338693    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012f6859-784b-4379-9979-6ed61173ce66-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:26.338697 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.338702    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012f6859-784b-4379-9979-6ed61173ce66-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:26.990936 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.990900    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8"
Nov 03 02:04:26.991131 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.990934    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74560ab4-6rtt8" event={"ID":"012f6859-784b-4379-9979-6ed61173ce66","Type":"ContainerDied","Data":"135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3"}
Nov 03 02:04:26.991131 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:26.990989    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135f908c651f31e492c8b623e570ff6ed18f821eeaff3fe23973339f0f3511e3"
Nov 03 02:04:33.556465 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.556423    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"]
Nov 03 02:04:33.556965 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.556874    4531 cpu_manager.go:414] "containerMap: removing container" podUID="012f6859-784b-4379-9979-6ed61173ce66" containerName="revoke-keys"
Nov 03 02:04:33.556965 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.556885    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012f6859-784b-4379-9979-6ed61173ce66" containerName="revoke-keys"
Nov 03 02:04:33.556965 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.556958    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="012f6859-784b-4379-9979-6ed61173ce66" containerName="revoke-keys"
Nov 03 02:04:33.559774 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.559754    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.564459 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.564433    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:33.564645 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.564501    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:33.567841 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.567773    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"]
Nov 03 02:04:33.610699 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.610660    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqfx\" (UniqueName: \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.610893 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.610714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.610893 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.610848    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.712370 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.712335    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqfx\" (UniqueName: \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.712549 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.712376    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.712549 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.712404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.712980 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.712954    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.714691 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.714660    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.721966 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.721943    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqfx\" (UniqueName: \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\") pod \"maas-api-revoke-sub-82a629e2-ffqv2\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:33.876640 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:33.876607    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:34.156015 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:34.155981    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"]
Nov 03 02:04:34.157286 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:34.157254    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6d728a_3479_4341_ae30_12b43122e516.slice/crio-bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f WatchSource:0}: Error finding container bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f: Status 404 returned error can't find the container with id bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f
Nov 03 02:04:35.046559 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:35.046514    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2" event={"ID":"7e6d728a-3479-4341-ae30-12b43122e516","Type":"ContainerStarted","Data":"9313135d8a1694b18e36fe95dcbf84c0cd44d634f3dec7c3a8b28f21c9c0709d"}
Nov 03 02:04:35.047094 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:35.046628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2" event={"ID":"7e6d728a-3479-4341-ae30-12b43122e516","Type":"ContainerStarted","Data":"bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f"}
Nov 03 02:04:35.068781 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:35.068716    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2" podStartSLOduration=2.068696609 podStartE2EDuration="2.068696609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:35.063811487 +0000 UTC m=+5020.277555161" watchObservedRunningTime="2026-11-03 02:04:35.068696609 +0000 UTC m=+5020.282440282"
Nov 03 02:04:36.092896 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:36.092399    4531 generic.go:356] "Generic (PLEG): container finished" podID="7e6d728a-3479-4341-ae30-12b43122e516" containerID="9313135d8a1694b18e36fe95dcbf84c0cd44d634f3dec7c3a8b28f21c9c0709d" exitCode=0
Nov 03 02:04:36.092896 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:36.092529    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2" event={"ID":"7e6d728a-3479-4341-ae30-12b43122e516","Type":"ContainerDied","Data":"9313135d8a1694b18e36fe95dcbf84c0cd44d634f3dec7c3a8b28f21c9c0709d"}
Nov 03 02:04:37.325502 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.325479    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:37.494291 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.494210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\") pod \"7e6d728a-3479-4341-ae30-12b43122e516\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") "
Nov 03 02:04:37.494490 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.494291    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\" (UniqueName: \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\") pod \"7e6d728a-3479-4341-ae30-12b43122e516\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") "
Nov 03 02:04:37.494490 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.494382    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\") pod \"7e6d728a-3479-4341-ae30-12b43122e516\" (UID: \"7e6d728a-3479-4341-ae30-12b43122e516\") "
Nov 03 02:04:37.494757 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.494726    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca" pod "7e6d728a-3479-4341-ae30-12b43122e516" (UID: "7e6d728a-3479-4341-ae30-12b43122e516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:37.496464 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.496430    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca" pod "7e6d728a-3479-4341-ae30-12b43122e516" (UID: "7e6d728a-3479-4341-ae30-12b43122e516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:37.496557 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.496468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx" pod "7e6d728a-3479-4341-ae30-12b43122e516" (UID: "7e6d728a-3479-4341-ae30-12b43122e516"). InnerVolumeSpecName "kube-api-access-4nqfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:37.595670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.595631    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:37.595670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.595663    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqfx\" (UniqueName: \"kubernetes.io/projected/7e6d728a-3479-4341-ae30-12b43122e516-kube-api-access-4nqfx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:37.595670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:37.595674    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d728a-3479-4341-ae30-12b43122e516-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:38.113237 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:38.113202    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2"
Nov 03 02:04:38.113237 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:38.113223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a629e2-ffqv2" event={"ID":"7e6d728a-3479-4341-ae30-12b43122e516","Type":"ContainerDied","Data":"bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f"}
Nov 03 02:04:38.113454 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:38.113273    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbd0eedea77394c3ef95f45730cb5247dd91b9e06ab0f47698ec2b958de7617f"
Nov 03 02:04:39.876624 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:04:39.876582    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:04:43.768368 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.768326    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"]
Nov 03 02:04:43.768946 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.768782    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7e6d728a-3479-4341-ae30-12b43122e516" containerName="revoke-keys"
Nov 03 02:04:43.768946 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.768794    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6d728a-3479-4341-ae30-12b43122e516" containerName="revoke-keys"
Nov 03 02:04:43.768946 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.768881    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6d728a-3479-4341-ae30-12b43122e516" containerName="revoke-keys"
Nov 03 02:04:43.772117 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.772096    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:43.777136 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.777110    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:43.777292 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.777206    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:43.779026 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.779002    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"]
Nov 03 02:04:43.957573 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.957534    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:43.957803 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.957671    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mpd\" (UniqueName: \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:43.957803 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:43.957726    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.058643 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.058547    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mpd\" (UniqueName: \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.058643 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.058600    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.058643 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.058639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.059305 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.059284    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.061108 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.061084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.067529 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.067503    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mpd\" (UniqueName: \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\") pod \"maas-api-revoke-sub-e529b4e2-x25td\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.095986 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.091037    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:44.368145 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:44.368118    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"]
Nov 03 02:04:44.370966 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:44.370927    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae053976_45d1_4b41_9be0_1b907ecdd1bf.slice/crio-7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843 WatchSource:0}: Error finding container 7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843: Status 404 returned error can't find the container with id 7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843
Nov 03 02:04:45.163957 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:45.163915    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td" event={"ID":"ae053976-45d1-4b41-9be0-1b907ecdd1bf","Type":"ContainerStarted","Data":"87dcb44925712ebc5862c68d984af01167082c03276a8037485ee756cb4990da"}
Nov 03 02:04:45.164630 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:45.164603    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td" event={"ID":"ae053976-45d1-4b41-9be0-1b907ecdd1bf","Type":"ContainerStarted","Data":"7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843"}
Nov 03 02:04:45.188737 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:45.188671    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td" podStartSLOduration=2.188652439 podStartE2EDuration="2.188652439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:45.183145815 +0000 UTC m=+5030.396889487" watchObservedRunningTime="2026-11-03 02:04:45.188652439 +0000 UTC m=+5030.402396113"
Nov 03 02:04:46.192127 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:46.192086    4531 generic.go:356] "Generic (PLEG): container finished" podID="ae053976-45d1-4b41-9be0-1b907ecdd1bf" containerID="87dcb44925712ebc5862c68d984af01167082c03276a8037485ee756cb4990da" exitCode=0
Nov 03 02:04:46.193429 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:46.193393    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td" event={"ID":"ae053976-45d1-4b41-9be0-1b907ecdd1bf","Type":"ContainerDied","Data":"87dcb44925712ebc5862c68d984af01167082c03276a8037485ee756cb4990da"}
Nov 03 02:04:47.436715 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.436684    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:47.507567 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.507462    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\" (UniqueName: \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\") pod \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") "
Nov 03 02:04:47.507567 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.507527    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\") pod \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") "
Nov 03 02:04:47.507834 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.507581    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\") pod \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\" (UID: \"ae053976-45d1-4b41-9be0-1b907ecdd1bf\") "
Nov 03 02:04:47.507904 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.507883    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca" pod "ae053976-45d1-4b41-9be0-1b907ecdd1bf" (UID: "ae053976-45d1-4b41-9be0-1b907ecdd1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:47.509685 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.509658    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca" pod "ae053976-45d1-4b41-9be0-1b907ecdd1bf" (UID: "ae053976-45d1-4b41-9be0-1b907ecdd1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:47.509808 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.509742    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd" pod "ae053976-45d1-4b41-9be0-1b907ecdd1bf" (UID: "ae053976-45d1-4b41-9be0-1b907ecdd1bf"). InnerVolumeSpecName "kube-api-access-h4mpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:47.608871 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.608830    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:47.608871 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.608861    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mpd\" (UniqueName: \"kubernetes.io/projected/ae053976-45d1-4b41-9be0-1b907ecdd1bf-kube-api-access-h4mpd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:47.608871 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:47.608871    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae053976-45d1-4b41-9be0-1b907ecdd1bf-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:48.210683 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:48.210652    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td"
Nov 03 02:04:48.210683 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:48.210661    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e529b4e2-x25td" event={"ID":"ae053976-45d1-4b41-9be0-1b907ecdd1bf","Type":"ContainerDied","Data":"7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843"}
Nov 03 02:04:48.210895 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:48.210714    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2dda5d7c5f74a75ba4abad47b8171a17cfc37245936e1c72cb698f453d6843"
Nov 03 02:04:53.985421 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.985386    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"]
Nov 03 02:04:53.985895 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.985880    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ae053976-45d1-4b41-9be0-1b907ecdd1bf" containerName="revoke-keys"
Nov 03 02:04:53.985943 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.985897    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae053976-45d1-4b41-9be0-1b907ecdd1bf" containerName="revoke-keys"
Nov 03 02:04:53.985985 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.985972    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae053976-45d1-4b41-9be0-1b907ecdd1bf" containerName="revoke-keys"
Nov 03 02:04:53.989364 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.989336    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:53.993708 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.993586    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:04:53.993708 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.993673    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:04:53.997406 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:53.997377    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"]
Nov 03 02:04:54.062078 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.062040    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hbj\" (UniqueName: \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.062286 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.062086    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.062286 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.062261    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.163575 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.163538    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.163738 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.163640    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hbj\" (UniqueName: \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.163738 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.163682    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.164438 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.164404    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.166021 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.165995    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.171690 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.171665    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hbj\" (UniqueName: \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\") pod \"maas-api-revoke-sub-5fcf32c9-z2r74\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.306368 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.306294    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:54.585758 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:54.585723    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"]
Nov 03 02:04:54.585944 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:04:54.585920    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d13afbe_f5fd_4d5f_8542_7e7c53185486.slice/crio-40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9 WatchSource:0}: Error finding container 40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9: Status 404 returned error can't find the container with id 40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9
Nov 03 02:04:55.255638 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:55.255527    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74" event={"ID":"6d13afbe-f5fd-4d5f-8542-7e7c53185486","Type":"ContainerStarted","Data":"9d37a9531e45cacca86b23940faa5e18b132ab0c29b60bd9c510548966b0b37e"}
Nov 03 02:04:55.255638 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:55.255604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74" event={"ID":"6d13afbe-f5fd-4d5f-8542-7e7c53185486","Type":"ContainerStarted","Data":"40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9"}
Nov 03 02:04:55.277495 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:55.277432    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74" podStartSLOduration=2.277413236 podStartE2EDuration="2.277413236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:55.273020274 +0000 UTC m=+5040.486763947" watchObservedRunningTime="2026-11-03 02:04:55.277413236 +0000 UTC m=+5040.491156910"
Nov 03 02:04:56.290184 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:56.289913    4531 generic.go:356] "Generic (PLEG): container finished" podID="6d13afbe-f5fd-4d5f-8542-7e7c53185486" containerID="9d37a9531e45cacca86b23940faa5e18b132ab0c29b60bd9c510548966b0b37e" exitCode=0
Nov 03 02:04:56.290184 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:56.290064    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74" event={"ID":"6d13afbe-f5fd-4d5f-8542-7e7c53185486","Type":"ContainerDied","Data":"9d37a9531e45cacca86b23940faa5e18b132ab0c29b60bd9c510548966b0b37e"}
Nov 03 02:04:57.622335 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.622315    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:57.710038 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.710008    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\") pod \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") "
Nov 03 02:04:57.710213 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.710126    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\") pod \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") "
Nov 03 02:04:57.710213 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.710193    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\" (UniqueName: \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\") pod \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\" (UID: \"6d13afbe-f5fd-4d5f-8542-7e7c53185486\") "
Nov 03 02:04:57.710442 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.710420    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca" pod "6d13afbe-f5fd-4d5f-8542-7e7c53185486" (UID: "6d13afbe-f5fd-4d5f-8542-7e7c53185486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:57.710560 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.710540    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:57.712542 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.712513    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca" pod "6d13afbe-f5fd-4d5f-8542-7e7c53185486" (UID: "6d13afbe-f5fd-4d5f-8542-7e7c53185486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:57.712724 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.712704    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj" pod "6d13afbe-f5fd-4d5f-8542-7e7c53185486" (UID: "6d13afbe-f5fd-4d5f-8542-7e7c53185486"). InnerVolumeSpecName "kube-api-access-t6hbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:57.811364 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.811328    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hbj\" (UniqueName: \"kubernetes.io/projected/6d13afbe-f5fd-4d5f-8542-7e7c53185486-kube-api-access-t6hbj\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:57.811364 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:57.811359    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d13afbe-f5fd-4d5f-8542-7e7c53185486-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:58.313517 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:58.313490    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74"
Nov 03 02:04:58.313517 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:58.313501    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf32c9-z2r74" event={"ID":"6d13afbe-f5fd-4d5f-8542-7e7c53185486","Type":"ContainerDied","Data":"40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9"}
Nov 03 02:04:58.313789 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:04:58.313551    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40874b32fabca0ace7f747bd1a05f2414d1b7b7de3626d80164295efc3ee92a9"
Nov 03 02:05:04.206475 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.206438    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"]
Nov 03 02:05:04.207022 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.206888    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6d13afbe-f5fd-4d5f-8542-7e7c53185486" containerName="revoke-keys"
Nov 03 02:05:04.207022 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.206901    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d13afbe-f5fd-4d5f-8542-7e7c53185486" containerName="revoke-keys"
Nov 03 02:05:04.207022 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.206979    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d13afbe-f5fd-4d5f-8542-7e7c53185486" containerName="revoke-keys"
Nov 03 02:05:04.210517 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.210492    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.215090 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.215065    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:04.215413 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.215227    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:04.219802 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.219778    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"]
Nov 03 02:05:04.273715 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.273668    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqzd\" (UniqueName: \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.273715 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.273713    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.273933 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.273819    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.375454 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.375393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqzd\" (UniqueName: \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.375454 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.375452    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.375763 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.375476    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.376085 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.376055    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.377936 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.377907    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.384493 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.384465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqzd\" (UniqueName: \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\") pod \"maas-api-revoke-sub-76da2167-g8t4f\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.527729 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.527647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:04.810772 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:04.810742    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"]
Nov 03 02:05:04.811828 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:04.811799    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1e9ff7_e4b6_4255_a6d0_5f72c34139a6.slice/crio-243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25 WatchSource:0}: Error finding container 243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25: Status 404 returned error can't find the container with id 243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25
Nov 03 02:05:05.367945 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:05.367899    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f" event={"ID":"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6","Type":"ContainerStarted","Data":"23b3bf7a57f54756bc547ad5d9def28780828ceba6d6efe17d2aa5047b8180e3"}
Nov 03 02:05:05.368526 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:05.367975    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f" event={"ID":"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6","Type":"ContainerStarted","Data":"243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25"}
Nov 03 02:05:05.393287 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:05.393174    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f" podStartSLOduration=1.39313961 podStartE2EDuration="1.39313961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:05.391512568 +0000 UTC m=+5050.605256251" watchObservedRunningTime="2026-11-03 02:05:05.39313961 +0000 UTC m=+5050.606883283"
Nov 03 02:05:06.401221 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:06.401180    4531 generic.go:356] "Generic (PLEG): container finished" podID="ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" containerID="23b3bf7a57f54756bc547ad5d9def28780828ceba6d6efe17d2aa5047b8180e3" exitCode=0
Nov 03 02:05:06.402605 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:06.402572    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f" event={"ID":"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6","Type":"ContainerDied","Data":"23b3bf7a57f54756bc547ad5d9def28780828ceba6d6efe17d2aa5047b8180e3"}
Nov 03 02:05:07.723622 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.723597    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:07.822403 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.822363    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\") pod \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") "
Nov 03 02:05:07.822604 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.822513    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\" (UniqueName: \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\") pod \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") "
Nov 03 02:05:07.822604 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.822547    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\") pod \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\" (UID: \"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6\") "
Nov 03 02:05:07.822977 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.822944    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca" pod "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" (UID: "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:07.824572 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.824541    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca" pod "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" (UID: "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:07.825091 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.825063    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd" pod "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" (UID: "ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6"). InnerVolumeSpecName "kube-api-access-6xqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:07.924055 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.924016    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:07.924055 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.924054    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqzd\" (UniqueName: \"kubernetes.io/projected/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-kube-api-access-6xqzd\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:07.924055 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:07.924065    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:08.415462 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:08.415432    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f"
Nov 03 02:05:08.415462 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:08.415454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76da2167-g8t4f" event={"ID":"ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6","Type":"ContainerDied","Data":"243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25"}
Nov 03 02:05:08.415691 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:08.415497    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243c843567b7506b95e258492163383b618fa1877742ccdad24c0348d77b8d25"
Nov 03 02:05:09.876810 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:05:09.876775    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:05:14.407981 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.407940    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"]
Nov 03 02:05:14.408431 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.408416    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" containerName="revoke-keys"
Nov 03 02:05:14.408481 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.408431    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" containerName="revoke-keys"
Nov 03 02:05:14.408518 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.408496    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6" containerName="revoke-keys"
Nov 03 02:05:14.411738 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.411721    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.422733 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.422707    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:14.422733 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.422707    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:14.427204 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.427137    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"]
Nov 03 02:05:14.485864 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.485805    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.485864 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.485872    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.486078 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.485952    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpjx\" (UniqueName: \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.586794 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.586747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.586794 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.586787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.587051 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.586819    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpjx\" (UniqueName: \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.587640 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.587613    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.589209 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.589189    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.594977 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.594947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpjx\" (UniqueName: \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\") pod \"maas-api-revoke-sub-bd909901-fxfpn\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:14.729670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:14.729585    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:15.008927 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:15.008892    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"]
Nov 03 02:05:15.010584 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:15.010558    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b3cf1c_7c81_4e2c_b843_6500690f1672.slice/crio-b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e WatchSource:0}: Error finding container b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e: Status 404 returned error can't find the container with id b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e
Nov 03 02:05:15.463928 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:15.463890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn" event={"ID":"25b3cf1c-7c81-4e2c-b843-6500690f1672","Type":"ContainerStarted","Data":"deed63912197ac1c20c57b4781e6635414c889c95928ffa400abb7421aabd113"}
Nov 03 02:05:15.464408 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:15.463946    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn" event={"ID":"25b3cf1c-7c81-4e2c-b843-6500690f1672","Type":"ContainerStarted","Data":"b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e"}
Nov 03 02:05:15.480538 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:15.480483    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn" podStartSLOduration=1.480468688 podStartE2EDuration="1.480468688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:15.47939062 +0000 UTC m=+5060.693134293" watchObservedRunningTime="2026-11-03 02:05:15.480468688 +0000 UTC m=+5060.694212357"
Nov 03 02:05:16.477406 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:16.477365    4531 generic.go:356] "Generic (PLEG): container finished" podID="25b3cf1c-7c81-4e2c-b843-6500690f1672" containerID="deed63912197ac1c20c57b4781e6635414c889c95928ffa400abb7421aabd113" exitCode=0
Nov 03 02:05:16.478128 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:16.478097    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn" event={"ID":"25b3cf1c-7c81-4e2c-b843-6500690f1672","Type":"ContainerDied","Data":"deed63912197ac1c20c57b4781e6635414c889c95928ffa400abb7421aabd113"}
Nov 03 02:05:17.820660 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.820634    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:17.991185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.990488    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\" (UniqueName: \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\") pod \"25b3cf1c-7c81-4e2c-b843-6500690f1672\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") "
Nov 03 02:05:17.991185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.990591    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\") pod \"25b3cf1c-7c81-4e2c-b843-6500690f1672\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") "
Nov 03 02:05:17.991185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.990685    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\") pod \"25b3cf1c-7c81-4e2c-b843-6500690f1672\" (UID: \"25b3cf1c-7c81-4e2c-b843-6500690f1672\") "
Nov 03 02:05:17.996954 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.996669    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca" pod "25b3cf1c-7c81-4e2c-b843-6500690f1672" (UID: "25b3cf1c-7c81-4e2c-b843-6500690f1672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:17.999419 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:17.999392    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx" pod "25b3cf1c-7c81-4e2c-b843-6500690f1672" (UID: "25b3cf1c-7c81-4e2c-b843-6500690f1672"). InnerVolumeSpecName "kube-api-access-wmpjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:18.006328 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.006296    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca" pod "25b3cf1c-7c81-4e2c-b843-6500690f1672" (UID: "25b3cf1c-7c81-4e2c-b843-6500690f1672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:18.091888 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.091796    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:18.091888 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.091842    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpjx\" (UniqueName: \"kubernetes.io/projected/25b3cf1c-7c81-4e2c-b843-6500690f1672-kube-api-access-wmpjx\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:18.091888 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.091859    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b3cf1c-7c81-4e2c-b843-6500690f1672-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:18.505224 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.505137    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn"
Nov 03 02:05:18.505224 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.505189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd909901-fxfpn" event={"ID":"25b3cf1c-7c81-4e2c-b843-6500690f1672","Type":"ContainerDied","Data":"b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e"}
Nov 03 02:05:18.505426 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:18.505231    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ff3225b235dae40a3b1e939e6308324e1aef26724a5fe37c6a9926760fe68e"
Nov 03 02:05:24.626909 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.626873    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"]
Nov 03 02:05:24.627484 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.627465    4531 cpu_manager.go:414] "containerMap: removing container" podUID="25b3cf1c-7c81-4e2c-b843-6500690f1672" containerName="revoke-keys"
Nov 03 02:05:24.627484 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.627485    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b3cf1c-7c81-4e2c-b843-6500690f1672" containerName="revoke-keys"
Nov 03 02:05:24.627630 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.627566    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b3cf1c-7c81-4e2c-b843-6500690f1672" containerName="revoke-keys"
Nov 03 02:05:24.631029 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.631007    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.636001 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.635968    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:24.636153 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.636113    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:24.642442 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.642392    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"]
Nov 03 02:05:24.657303 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.657272    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.657464 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.657415    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.657464 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.657459    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsnm\" (UniqueName: \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.757978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.757945    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.758193 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.758015    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.758193 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.758044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsnm\" (UniqueName: \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.758638 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.758614    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.760409 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.760386    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.766331 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.766309    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsnm\" (UniqueName: \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\") pod \"maas-api-revoke-sub-61d3bdbb-l8wrr\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:24.946973 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:24.946894    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:25.233976 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:25.233941    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"]
Nov 03 02:05:25.235557 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:25.235522    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9291402a_db3c_4b00_b527_4ab9340c7820.slice/crio-c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711 WatchSource:0}: Error finding container c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711: Status 404 returned error can't find the container with id c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711
Nov 03 02:05:25.552025 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:25.551991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr" event={"ID":"9291402a-db3c-4b00-b527-4ab9340c7820","Type":"ContainerStarted","Data":"9c194abd04279af538479a7c54e8875e1ed04c042fb43815f1c7ef9318123fa2"}
Nov 03 02:05:25.552224 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:25.552058    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr" event={"ID":"9291402a-db3c-4b00-b527-4ab9340c7820","Type":"ContainerStarted","Data":"c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711"}
Nov 03 02:05:25.569713 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:25.569663    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr" podStartSLOduration=1.569649275 podStartE2EDuration="1.569649275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:25.565978424 +0000 UTC m=+5070.779722097" watchObservedRunningTime="2026-11-03 02:05:25.569649275 +0000 UTC m=+5070.783392949"
Nov 03 02:05:26.564838 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:26.564737    4531 generic.go:356] "Generic (PLEG): container finished" podID="9291402a-db3c-4b00-b527-4ab9340c7820" containerID="9c194abd04279af538479a7c54e8875e1ed04c042fb43815f1c7ef9318123fa2" exitCode=0
Nov 03 02:05:26.564838 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:26.564802    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr" event={"ID":"9291402a-db3c-4b00-b527-4ab9340c7820","Type":"ContainerDied","Data":"9c194abd04279af538479a7c54e8875e1ed04c042fb43815f1c7ef9318123fa2"}
Nov 03 02:05:28.010981 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.010943    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:28.119253 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.119224    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\") pod \"9291402a-db3c-4b00-b527-4ab9340c7820\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") "
Nov 03 02:05:28.119492 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.119478    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\" (UniqueName: \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\") pod \"9291402a-db3c-4b00-b527-4ab9340c7820\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") "
Nov 03 02:05:28.119603 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.119592    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\") pod \"9291402a-db3c-4b00-b527-4ab9340c7820\" (UID: \"9291402a-db3c-4b00-b527-4ab9340c7820\") "
Nov 03 02:05:28.119732 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.119626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca" pod "9291402a-db3c-4b00-b527-4ab9340c7820" (UID: "9291402a-db3c-4b00-b527-4ab9340c7820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:28.119973 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.119955    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:28.121550 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.121531    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm" pod "9291402a-db3c-4b00-b527-4ab9340c7820" (UID: "9291402a-db3c-4b00-b527-4ab9340c7820"). InnerVolumeSpecName "kube-api-access-cxsnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:28.122857 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.122839    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca" pod "9291402a-db3c-4b00-b527-4ab9340c7820" (UID: "9291402a-db3c-4b00-b527-4ab9340c7820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:28.221356 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.221267    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsnm\" (UniqueName: \"kubernetes.io/projected/9291402a-db3c-4b00-b527-4ab9340c7820-kube-api-access-cxsnm\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:28.221356 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.221303    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9291402a-db3c-4b00-b527-4ab9340c7820-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:28.598134 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.598105    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr"
Nov 03 02:05:28.598310 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.598138    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d3bdbb-l8wrr" event={"ID":"9291402a-db3c-4b00-b527-4ab9340c7820","Type":"ContainerDied","Data":"c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711"}
Nov 03 02:05:28.598310 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:28.598206    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5f6f9dcd8e6667d617a4e959fa0ab287e883fd9c89a0165ce92eae337e79711"
Nov 03 02:05:34.854605 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.854563    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"]
Nov 03 02:05:34.855046 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.855007    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9291402a-db3c-4b00-b527-4ab9340c7820" containerName="revoke-keys"
Nov 03 02:05:34.855046 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.855019    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9291402a-db3c-4b00-b527-4ab9340c7820" containerName="revoke-keys"
Nov 03 02:05:34.855146 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.855122    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9291402a-db3c-4b00-b527-4ab9340c7820" containerName="revoke-keys"
Nov 03 02:05:34.858597 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.858575    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:34.864138 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.864115    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:34.864353 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.864133    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:34.870908 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.870881    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"]
Nov 03 02:05:34.991137 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.991098    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dlp\" (UniqueName: \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:34.991349 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.991184    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:34.991583 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:34.991544    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.092978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.092920    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.092978 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.092989    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dlp\" (UniqueName: \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.093414 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.093014    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.093644 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.093623    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.095422 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.095397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.100965 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.100942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dlp\" (UniqueName: \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\") pod \"maas-api-revoke-sub-3125f542-4bdwc\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.175810 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.175735    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:35.458512 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:35.458469    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9efb5_5d70_4c66_aad0_fed4f08b9d12.slice/crio-3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7 WatchSource:0}: Error finding container 3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7: Status 404 returned error can't find the container with id 3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7
Nov 03 02:05:35.459036 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.459008    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"]
Nov 03 02:05:35.649828 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:35.649795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc" event={"ID":"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12","Type":"ContainerStarted","Data":"3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7"}
Nov 03 02:05:36.658528 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:36.658316    4531 generic.go:356] "Generic (PLEG): container finished" podID="d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" containerID="a009165ab066ff0db605d23e1f67622241e1edf08ce04c0751bc978581482913" exitCode=0
Nov 03 02:05:36.658528 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:36.658502    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc" event={"ID":"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12","Type":"ContainerDied","Data":"a009165ab066ff0db605d23e1f67622241e1edf08ce04c0751bc978581482913"}
Nov 03 02:05:38.015093 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.015061    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:38.123429 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.123394    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\") pod \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") "
Nov 03 02:05:38.123650 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.123463    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\") pod \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") "
Nov 03 02:05:38.123650 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.123613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\" (UniqueName: \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\") pod \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\" (UID: \"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12\") "
Nov 03 02:05:38.124154 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.124124    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca" pod "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" (UID: "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:38.126003 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.125978    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca" pod "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" (UID: "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:38.126117 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.126086    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp" pod "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" (UID: "d2d9efb5-5d70-4c66-aad0-fed4f08b9d12"). InnerVolumeSpecName "kube-api-access-q5dlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:38.225068 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.225031    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:38.225068 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.225061    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:38.225068 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.225071    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dlp\" (UniqueName: \"kubernetes.io/projected/d2d9efb5-5d70-4c66-aad0-fed4f08b9d12-kube-api-access-q5dlp\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:38.684237 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.684203    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc"
Nov 03 02:05:38.684418 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.684203    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3125f542-4bdwc" event={"ID":"d2d9efb5-5d70-4c66-aad0-fed4f08b9d12","Type":"ContainerDied","Data":"3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7"}
Nov 03 02:05:38.684418 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:38.684356    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed846b2341285aecd5d75a716c0e3a667ae1a5cd2647e96f4a7067919a8dbe7"
Nov 03 02:05:39.876897 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:05:39.876859    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:05:45.104324 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.104287    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"]
Nov 03 02:05:45.104842 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.104777    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" containerName="revoke-keys"
Nov 03 02:05:45.104842 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.104792    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" containerName="revoke-keys"
Nov 03 02:05:45.104984 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.104885    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d9efb5-5d70-4c66-aad0-fed4f08b9d12" containerName="revoke-keys"
Nov 03 02:05:45.108225 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.108203    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.112654 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.112628    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:45.112902 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.112873    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:45.118814 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.118782    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"]
Nov 03 02:05:45.193101 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.193067    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.193297 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.193124    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6x7h\" (UniqueName: \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.193353 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.193297    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.294323 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.294288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.294533 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.294357    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.294533 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.294423    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6x7h\" (UniqueName: \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.295050 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.295030    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.296639 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.296617    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.303505 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.303449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6x7h\" (UniqueName: \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\") pod \"maas-api-revoke-sub-925f0a15-kh7sm\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.425896 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.425809    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:45.703723 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.703696    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"]
Nov 03 02:05:45.705605 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:45.705570    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7922624_878e_459b_ba7f_3d6df801ff3e.slice/crio-20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0 WatchSource:0}: Error finding container 20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0: Status 404 returned error can't find the container with id 20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0
Nov 03 02:05:45.728069 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:45.728041    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm" event={"ID":"b7922624-878e-459b-ba7f-3d6df801ff3e","Type":"ContainerStarted","Data":"20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0"}
Nov 03 02:05:46.740656 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:46.740611    4531 generic.go:356] "Generic (PLEG): container finished" podID="b7922624-878e-459b-ba7f-3d6df801ff3e" containerID="e07ef4fda1f0b08038015ecf432c2f4f452e834b8b82c883aa99833460eeda96" exitCode=0
Nov 03 02:05:46.741153 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:46.740813    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm" event={"ID":"b7922624-878e-459b-ba7f-3d6df801ff3e","Type":"ContainerDied","Data":"e07ef4fda1f0b08038015ecf432c2f4f452e834b8b82c883aa99833460eeda96"}
Nov 03 02:05:48.158188 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.157350    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:48.233185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.230467    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\" (UniqueName: \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\") pod \"b7922624-878e-459b-ba7f-3d6df801ff3e\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") "
Nov 03 02:05:48.233185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.230561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\") pod \"b7922624-878e-459b-ba7f-3d6df801ff3e\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") "
Nov 03 02:05:48.233185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.230688    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\") pod \"b7922624-878e-459b-ba7f-3d6df801ff3e\" (UID: \"b7922624-878e-459b-ba7f-3d6df801ff3e\") "
Nov 03 02:05:48.253184 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.245377    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca" pod "b7922624-878e-459b-ba7f-3d6df801ff3e" (UID: "b7922624-878e-459b-ba7f-3d6df801ff3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:48.275182 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.273657    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca" pod "b7922624-878e-459b-ba7f-3d6df801ff3e" (UID: "b7922624-878e-459b-ba7f-3d6df801ff3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:48.282181 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.279289    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h" pod "b7922624-878e-459b-ba7f-3d6df801ff3e" (UID: "b7922624-878e-459b-ba7f-3d6df801ff3e"). InnerVolumeSpecName "kube-api-access-c6x7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:48.336190 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.333750    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:48.336190 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.333794    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6x7h\" (UniqueName: \"kubernetes.io/projected/b7922624-878e-459b-ba7f-3d6df801ff3e-kube-api-access-c6x7h\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:48.336190 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.333811    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7922624-878e-459b-ba7f-3d6df801ff3e-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:48.803762 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.803723    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm"
Nov 03 02:05:48.803762 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.803750    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925f0a15-kh7sm" event={"ID":"b7922624-878e-459b-ba7f-3d6df801ff3e","Type":"ContainerDied","Data":"20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0"}
Nov 03 02:05:48.803986 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:48.803802    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20be866f5e40bed20f781bb71da458062c66d0a693f974d38bd7bd5ddb9980e0"
Nov 03 02:05:55.310824 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.310774    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"]
Nov 03 02:05:55.311651 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.311552    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b7922624-878e-459b-ba7f-3d6df801ff3e" containerName="revoke-keys"
Nov 03 02:05:55.311651 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.311574    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7922624-878e-459b-ba7f-3d6df801ff3e" containerName="revoke-keys"
Nov 03 02:05:55.311785 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.311678    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7922624-878e-459b-ba7f-3d6df801ff3e" containerName="revoke-keys"
Nov 03 02:05:55.315344 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.315320    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.321523 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.321497    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:05:55.321663 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.321497    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-swfcr\""
Nov 03 02:05:55.323874 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.323847    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"]
Nov 03 02:05:55.413268 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.413227    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.413457 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.413276    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws27d\" (UniqueName: \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.413457 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.413367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.514545 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.514502    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.514545 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.514549    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws27d\" (UniqueName: \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.514838 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.514577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.515355 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.515330    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.516963 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.516941    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.522756 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.522733    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws27d\" (UniqueName: \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\") pod \"maas-api-revoke-sub-ad024866-2c9rb\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.636418 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.636382    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:55.927534 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:55.927498    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"]
Nov 03 02:05:55.928065 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:05:55.928037    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f974643_296a_486b_8f9d_8a2dc3300cb9.slice/crio-a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765 WatchSource:0}: Error finding container a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765: Status 404 returned error can't find the container with id a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765
Nov 03 02:05:56.891193 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:56.889847    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb" event={"ID":"4f974643-296a-486b-8f9d-8a2dc3300cb9","Type":"ContainerStarted","Data":"f86fb3e9f4de994877feefe6a066df537b6a308b85d4651aac9ae4c40f1edbc0"}
Nov 03 02:05:56.891193 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:56.889922    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb" event={"ID":"4f974643-296a-486b-8f9d-8a2dc3300cb9","Type":"ContainerStarted","Data":"a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765"}
Nov 03 02:05:56.919148 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:56.919092    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb" podStartSLOduration=1.9190772109999998 podStartE2EDuration="1.919077211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:56.917261243 +0000 UTC m=+5102.131004916" watchObservedRunningTime="2026-11-03 02:05:56.919077211 +0000 UTC m=+5102.132820882"
Nov 03 02:05:57.902185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:57.901611    4531 generic.go:356] "Generic (PLEG): container finished" podID="4f974643-296a-486b-8f9d-8a2dc3300cb9" containerID="f86fb3e9f4de994877feefe6a066df537b6a308b85d4651aac9ae4c40f1edbc0" exitCode=0
Nov 03 02:05:57.902185 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:57.901763    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb" event={"ID":"4f974643-296a-486b-8f9d-8a2dc3300cb9","Type":"ContainerDied","Data":"f86fb3e9f4de994877feefe6a066df537b6a308b85d4651aac9ae4c40f1edbc0"}
Nov 03 02:05:59.037875 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.037850    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:59.213309 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.213209    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\" (UniqueName: \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\") pod \"4f974643-296a-486b-8f9d-8a2dc3300cb9\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") "
Nov 03 02:05:59.213309 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.213307    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\") pod \"4f974643-296a-486b-8f9d-8a2dc3300cb9\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") "
Nov 03 02:05:59.213549 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.213339    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\") pod \"4f974643-296a-486b-8f9d-8a2dc3300cb9\" (UID: \"4f974643-296a-486b-8f9d-8a2dc3300cb9\") "
Nov 03 02:05:59.213707 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.213685    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca" pod "4f974643-296a-486b-8f9d-8a2dc3300cb9" (UID: "4f974643-296a-486b-8f9d-8a2dc3300cb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:05:59.215350 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.215325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca" pod "4f974643-296a-486b-8f9d-8a2dc3300cb9" (UID: "4f974643-296a-486b-8f9d-8a2dc3300cb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:05:59.215474 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.215456    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d" pod "4f974643-296a-486b-8f9d-8a2dc3300cb9" (UID: "4f974643-296a-486b-8f9d-8a2dc3300cb9"). InnerVolumeSpecName "kube-api-access-ws27d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:05:59.314555 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.314512    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws27d\" (UniqueName: \"kubernetes.io/projected/4f974643-296a-486b-8f9d-8a2dc3300cb9-kube-api-access-ws27d\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:59.314555 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.314545    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-openshift-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:59.314555 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.314556    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f974643-296a-486b-8f9d-8a2dc3300cb9-maas-api-xks-ca\") on node \"ip-10-0-136-112.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 02:05:59.914475 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.914450    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb"
Nov 03 02:05:59.914651 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.914444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad024866-2c9rb" event={"ID":"4f974643-296a-486b-8f9d-8a2dc3300cb9","Type":"ContainerDied","Data":"a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765"}
Nov 03 02:05:59.914651 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:05:59.914568    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a452f6181b745ef30dcace75ab6848676cdafa8dc0a1cde064cbb3ac0da74765"
Nov 03 02:06:09.882976 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:06:09.882933    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:06:27.589262 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:27.589225    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/1.log"
Nov 03 02:06:27.771944 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:27.771913    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-n922j_29693553-4e1c-4e36-aad0-a1360b84ba72/manager/0.log"
Nov 03 02:06:27.967123 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:27.967025    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kkdsj_8d36a864-53d0-4e75-b3bc-867fc1d42a89/manager/0.log"
Nov 03 02:06:28.427105 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:28.427079    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knbqk_49fee83f-2189-40b6-8559-022d6cc91770/server/0.log"
Nov 03 02:06:28.613594 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:28.613565    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ctxbc_11772d42-58e7-41ad-8adb-28220a88c2f9/manager/0.log"
Nov 03 02:06:28.830393 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:28.830362    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzws2_2432b11a-588d-42c8-9698-1787ca42ed40/proxy/0.log"
Nov 03 02:06:29.385218 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:29.385185    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/0.log"
Nov 03 02:06:30.273817 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:30.273787    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-25c1e8b5-lgpxl_ac1323b7-f732-45f6-80af-be127525fbcc/revoke-keys/0.log"
Nov 03 02:06:30.466567 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:30.466538    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3125f542-4bdwc_d2d9efb5-5d70-4c66-aad0-fed4f08b9d12/revoke-keys/0.log"
Nov 03 02:06:30.638976 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:30.638950    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-33397942-w24lb_c840156d-b881-4e59-9d76-e881a02147f6/revoke-keys/0.log"
Nov 03 02:06:30.814765 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:30.814738    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4bce2bf8-z92cm_8a54bea8-6445-4634-9c1b-091a34d15c06/revoke-keys/0.log"
Nov 03 02:06:30.989798 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:30.989725    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5fcf32c9-z2r74_6d13afbe-f5fd-4d5f-8542-7e7c53185486/revoke-keys/0.log"
Nov 03 02:06:31.173559 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:31.173528    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-61d3bdbb-l8wrr_9291402a-db3c-4b00-b527-4ab9340c7820/revoke-keys/0.log"
Nov 03 02:06:31.349755 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:31.349720    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-74560ab4-6rtt8_012f6859-784b-4379-9979-6ed61173ce66/revoke-keys/0.log"
Nov 03 02:06:31.524549 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:31.524511    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-76da2167-g8t4f_ae1e9ff7-e4b6-4255-a6d0-5f72c34139a6/revoke-keys/0.log"
Nov 03 02:06:31.715615 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:31.715539    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-81e14ee0-hknxh_3b6e09da-340a-4189-ba4c-4b806232cc65/revoke-keys/0.log"
Nov 03 02:06:31.890050 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:31.890023    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-82a629e2-ffqv2_7e6d728a-3479-4341-ae30-12b43122e516/revoke-keys/0.log"
Nov 03 02:06:32.090867 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:32.090846    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8d440fa1-l2lrp_4bddff25-15ae-43bb-8846-097b492619a9/revoke-keys/0.log"
Nov 03 02:06:32.281011 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:32.280984    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-925f0a15-kh7sm_b7922624-878e-459b-ba7f-3d6df801ff3e/revoke-keys/0.log"
Nov 03 02:06:32.663327 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:32.663302    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ad024866-2c9rb_4f974643-296a-486b-8f9d-8a2dc3300cb9/revoke-keys/0.log"
Nov 03 02:06:32.828844 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:32.828816    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bd909901-fxfpn_25b3cf1c-7c81-4e2c-b843-6500690f1672/revoke-keys/0.log"
Nov 03 02:06:32.999399 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:32.999331    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c0af0409-h9wrb_9ea15f1c-1d1b-4299-bf6a-530425ace9ce/revoke-keys/0.log"
Nov 03 02:06:33.177739 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:33.177714    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e529b4e2-x25td_ae053976-45d1-4b41-9be0-1b907ecdd1bf/revoke-keys/0.log"
Nov 03 02:06:33.377384 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:33.377351    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e8505b62-bgsxn_f522cd0e-7ac6-4271-a30c-5f1f825c9ea3/revoke-keys/0.log"
Nov 03 02:06:33.572363 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:33.572335    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f979a8c5-kr4cp_594e95cc-d9c4-4fe4-ad0f-334dbdc193f8/revoke-keys/0.log"
Nov 03 02:06:33.759430 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:33.759361    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kt22v_074e788b-3f4c-4b98-a302-c67107a603bc/postgres/0.log"
Nov 03 02:06:35.853374 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:35.853339    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c9svn_d65cb9f3-5969-41c3-8f50-42bf0f002c72/manager/0.log"
Nov 03 02:06:36.029763 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:36.029733    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/1.log"
Nov 03 02:06:36.374670 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:36.374637    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-btqdp_20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e/manager/0.log"
Nov 03 02:06:36.755138 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:36.755067    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-szqjl_dbf70c07-8ae7-4884-933c-17d7dccf6806/manager/0.log"
Nov 03 02:06:37.335418 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:37.335386    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/0.log"
Nov 03 02:06:37.842151 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:37.842121    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865gxcxg_65a72581-3f3f-4ff5-b75c-c97eba306705/istio-proxy/0.log"
Nov 03 02:06:38.604018 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:38.603985    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-46hnd_37b1042d-911e-4e83-b83d-dbb8096646d3/istio-proxy/0.log"
Nov 03 02:06:39.177880 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:39.177847    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-d852p_c9342480-831c-4417-8711-a69e79b817a6/istio-proxy/0.log"
Nov 03 02:06:39.575329 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:39.575303    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-ptjsj_51ac183a-e533-4d5f-91c4-94d05308d76a/istio-proxy/0.log"
Nov 03 02:06:39.882374 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:06:39.882294    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:06:40.509027 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:40.508985    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n_1c6afe96-ef02-4368-b078-6ec4b0e0e1ff/istio-proxy/0.log"
Nov 03 02:06:53.585004 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.584965    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-29h5c/must-gather-rh4jn"]
Nov 03 02:06:53.585540 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.585418    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4f974643-296a-486b-8f9d-8a2dc3300cb9" containerName="revoke-keys"
Nov 03 02:06:53.585540 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.585432    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f974643-296a-486b-8f9d-8a2dc3300cb9" containerName="revoke-keys"
Nov 03 02:06:53.585540 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.585521    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f974643-296a-486b-8f9d-8a2dc3300cb9" containerName="revoke-keys"
Nov 03 02:06:53.589093 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.589071    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.596316 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.596287    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-29h5c\"/\"default-dockercfg-h6wwm\""
Nov 03 02:06:53.596468 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.596391    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29h5c\"/\"kube-root-ca.crt\""
Nov 03 02:06:53.597475 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.596811    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-29h5c\"/\"openshift-service-ca.crt\""
Nov 03 02:06:53.597604 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.597584    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29h5c/must-gather-rh4jn"]
Nov 03 02:06:53.644431 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.644401    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55p69\" (UniqueName: \"kubernetes.io/projected/640fa617-a199-4f29-a2cf-9743c0dee541-kube-api-access-55p69\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.644588 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.644441    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/640fa617-a199-4f29-a2cf-9743c0dee541-must-gather-output\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.745858 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.745811    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55p69\" (UniqueName: \"kubernetes.io/projected/640fa617-a199-4f29-a2cf-9743c0dee541-kube-api-access-55p69\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.746064 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.745882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/640fa617-a199-4f29-a2cf-9743c0dee541-must-gather-output\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.746293 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.746273    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/640fa617-a199-4f29-a2cf-9743c0dee541-must-gather-output\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.754422 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.754402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55p69\" (UniqueName: \"kubernetes.io/projected/640fa617-a199-4f29-a2cf-9743c0dee541-kube-api-access-55p69\") pod \"must-gather-rh4jn\" (UID: \"640fa617-a199-4f29-a2cf-9743c0dee541\") " pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:53.905964 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:53.905885    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-29h5c/must-gather-rh4jn"
Nov 03 02:06:54.185979 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:54.185949    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-29h5c/must-gather-rh4jn"]
Nov 03 02:06:54.186356 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:06:54.186334    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod640fa617_a199_4f29_a2cf_9743c0dee541.slice/crio-45f801809bc7ccef6c81bc5eb690e197cb70a6b920bd3303f8309c09ffd22eb8 WatchSource:0}: Error finding container 45f801809bc7ccef6c81bc5eb690e197cb70a6b920bd3303f8309c09ffd22eb8: Status 404 returned error can't find the container with id 45f801809bc7ccef6c81bc5eb690e197cb70a6b920bd3303f8309c09ffd22eb8
Nov 03 02:06:54.188142 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:54.188125    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:06:54.271524 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:54.271493    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29h5c/must-gather-rh4jn" event={"ID":"640fa617-a199-4f29-a2cf-9743c0dee541","Type":"ContainerStarted","Data":"45f801809bc7ccef6c81bc5eb690e197cb70a6b920bd3303f8309c09ffd22eb8"}
Nov 03 02:06:57.621565 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:57.621530    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vg7f_04c2c55c-c14f-438b-b025-0ccf42d1d253/global-pull-secret-syncer/0.log"
Nov 03 02:06:57.875767 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:57.875682    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cbkz7_3c45381d-5957-4d81-847f-5e3ad1b1780c/konnectivity-agent/0.log"
Nov 03 02:06:57.994613 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:06:57.994580    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-112.us-east-2.compute.internal_4ecdbd9fc26c68793bebc9a7943385b7/haproxy/0.log"
Nov 03 02:07:00.329840 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:00.329799    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29h5c/must-gather-rh4jn" event={"ID":"640fa617-a199-4f29-a2cf-9743c0dee541","Type":"ContainerStarted","Data":"afaba9bfdb73d74bd750ad35f31f9b8bf71c4ede3f84ddbcb07c81e3cb331c78"}
Nov 03 02:07:00.330477 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:00.330441    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-29h5c/must-gather-rh4jn" event={"ID":"640fa617-a199-4f29-a2cf-9743c0dee541","Type":"ContainerStarted","Data":"4f456841777ed5fa21c9342a575eb9c67c2051e1e94c383f0b6e529243ae7a57"}
Nov 03 02:07:00.349490 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:00.349440    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-29h5c/must-gather-rh4jn" podStartSLOduration=1.537328362 podStartE2EDuration="7.34942677s" totalImagesPullingTime="5.812098408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:00.346146576 +0000 UTC m=+5165.559890247" watchObservedRunningTime="2026-11-03 02:07:00.34942677 +0000 UTC m=+5165.563170466"
Nov 03 02:07:01.220240 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:01.220207    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/1.log"
Nov 03 02:07:01.220415 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:01.220278    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/0.log"
Nov 03 02:07:01.291760 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:01.291725    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-n922j_29693553-4e1c-4e36-aad0-a1360b84ba72/manager/0.log"
Nov 03 02:07:02.217360 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:02.217322    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kkdsj_8d36a864-53d0-4e75-b3bc-867fc1d42a89/manager/0.log"
Nov 03 02:07:02.307525 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:02.307495    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knbqk_49fee83f-2189-40b6-8559-022d6cc91770/server/0.log"
Nov 03 02:07:03.484442 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:03.484401    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ctxbc_11772d42-58e7-41ad-8adb-28220a88c2f9/manager/0.log"
Nov 03 02:07:03.507117 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:03.507090    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzws2_2432b11a-588d-42c8-9698-1787ca42ed40/proxy/0.log"
Nov 03 02:07:04.330528 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:04.330495    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/1.log"
Nov 03 02:07:04.330528 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:04.330518    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/0.log"
Nov 03 02:07:04.409151 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:04.409106    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-n922j_29693553-4e1c-4e36-aad0-a1360b84ba72/manager/0.log"
Nov 03 02:07:05.025841 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.025807    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"]
Nov 03 02:07:05.029836 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.029807    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8505b62-bgsxn"]
Nov 03 02:07:05.356857 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.356819    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kkdsj_8d36a864-53d0-4e75-b3bc-867fc1d42a89/manager/0.log"
Nov 03 02:07:05.496663 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.496633    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knbqk_49fee83f-2189-40b6-8559-022d6cc91770/server/0.log"
Nov 03 02:07:05.776801 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.776723    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c9svn_d65cb9f3-5969-41c3-8f50-42bf0f002c72/manager/0.log"
Nov 03 02:07:05.815678 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.815647    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/1.log"
Nov 03 02:07:05.815846 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.815695    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/0.log"
Nov 03 02:07:05.918924 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:05.918886    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-btqdp_20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e/manager/0.log"
Nov 03 02:07:06.105250 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:06.105216    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-szqjl_dbf70c07-8ae7-4884-933c-17d7dccf6806/manager/0.log"
Nov 03 02:07:06.791872 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:06.791738    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ctxbc_11772d42-58e7-41ad-8adb-28220a88c2f9/manager/0.log"
Nov 03 02:07:06.873282 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:06.873244    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzws2_2432b11a-588d-42c8-9698-1787ca42ed40/proxy/0.log"
Nov 03 02:07:06.909546 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:06.909502    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f522cd0e-7ac6-4271-a30c-5f1f825c9ea3" path="/var/lib/kubelet/pods/f522cd0e-7ac6-4271-a30c-5f1f825c9ea3/volumes"
Nov 03 02:07:07.773984 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.773958    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/1.log"
Nov 03 02:07:07.774397 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.774025    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mpgsb_61eb856b-5461-45bb-9126-c9d33d325ae7/manager/0.log"
Nov 03 02:07:07.820925 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.820891    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/alertmanager/0.log"
Nov 03 02:07:07.847739 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.847713    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-n922j_29693553-4e1c-4e36-aad0-a1360b84ba72/manager/0.log"
Nov 03 02:07:07.858932 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.858908    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/config-reloader/0.log"
Nov 03 02:07:07.890172 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.890140    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/kube-rbac-proxy-web/0.log"
Nov 03 02:07:07.916901 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.916874    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/kube-rbac-proxy/0.log"
Nov 03 02:07:07.957453 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.957426    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/kube-rbac-proxy-metric/0.log"
Nov 03 02:07:07.996630 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:07.996606    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/prom-label-proxy/0.log"
Nov 03 02:07:08.024925 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.024852    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c5afd5b-b33b-4a82-bc4c-b31e4c098be0/init-config-reloader/0.log"
Nov 03 02:07:08.116923 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.116883    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6n9rm_1c948d3d-b8cd-43c5-b111-dba302be2a5b/kube-state-metrics/0.log"
Nov 03 02:07:08.153627 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.153600    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6n9rm_1c948d3d-b8cd-43c5-b111-dba302be2a5b/kube-rbac-proxy-main/0.log"
Nov 03 02:07:08.183227 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.183196    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-6n9rm_1c948d3d-b8cd-43c5-b111-dba302be2a5b/kube-rbac-proxy-self/0.log"
Nov 03 02:07:08.216817 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.216787    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55f7ccd4c6-4tq9l_cf0dce65-0138-4e7d-8303-745ec180676a/metrics-server/0.log"
Nov 03 02:07:08.384753 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.384723    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6mk4l_da9d2dac-e9f8-436e-8ded-c7a33eeb380a/node-exporter/0.log"
Nov 03 02:07:08.415868 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.415834    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6mk4l_da9d2dac-e9f8-436e-8ded-c7a33eeb380a/kube-rbac-proxy/0.log"
Nov 03 02:07:08.443034 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.443003    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6mk4l_da9d2dac-e9f8-436e-8ded-c7a33eeb380a/init-textfile/0.log"
Nov 03 02:07:08.813083 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.812979    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-kkdsj_8d36a864-53d0-4e75-b3bc-867fc1d42a89/manager/0.log"
Nov 03 02:07:08.962782 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.962755    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knbqk_49fee83f-2189-40b6-8559-022d6cc91770/server/0.log"
Nov 03 02:07:08.970964 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.970936    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"]
Nov 03 02:07:08.979060 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.979037    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:08.983405 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.983381    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grd2d\"/\"default-dockercfg-wj6sf\""
Nov 03 02:07:08.984399 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.984370    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"]
Nov 03 02:07:08.986365 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.986338    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"openshift-service-ca.crt\""
Nov 03 02:07:08.986481 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:08.986440    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grd2d\"/\"kube-root-ca.crt\""
Nov 03 02:07:09.030625 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.030578    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b94879f6d-6gw74_23a3d848-e0a8-4002-b281-006764a14e42/telemeter-client/0.log"
Nov 03 02:07:09.057927 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.057895    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b94879f6d-6gw74_23a3d848-e0a8-4002-b281-006764a14e42/reload/0.log"
Nov 03 02:07:09.078769 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.078730    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzfn\" (UniqueName: \"kubernetes.io/projected/66e809e3-4e26-4e74-aa78-9173c5073237-kube-api-access-fgzfn\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.078951 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.078872    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-lib-modules\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.078951 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.078903    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-podres\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.079133 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.078945    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-sys\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.079133 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.079087    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-proc\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.093732 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.093705    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b94879f6d-6gw74_23a3d848-e0a8-4002-b281-006764a14e42/kube-rbac-proxy/0.log"
Nov 03 02:07:09.179625 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179584    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzfn\" (UniqueName: \"kubernetes.io/projected/66e809e3-4e26-4e74-aa78-9173c5073237-kube-api-access-fgzfn\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.179814 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-lib-modules\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.179814 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179718    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-podres\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.179814 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-sys\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.179814 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179806    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-lib-modules\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.180134 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179819    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-proc\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.180134 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-podres\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.180134 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179851    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-sys\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.180134 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.179871    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66e809e3-4e26-4e74-aa78-9173c5073237-proc\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.188972 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.188943    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzfn\" (UniqueName: \"kubernetes.io/projected/66e809e3-4e26-4e74-aa78-9173c5073237-kube-api-access-fgzfn\") pod \"perf-node-gather-daemonset-gk79x\" (UID: \"66e809e3-4e26-4e74-aa78-9173c5073237\") " pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.306327 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.306296    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:09.606459 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:09.606364    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"]
Nov 03 02:07:09.606775 ip-10-0-136-112 kubenswrapper[4531]: W1103 02:07:09.606744    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod66e809e3_4e26_4e74_aa78_9173c5073237.slice/crio-03f38df58414ef8490b0dcb43aac1ecdbc0a538ba63b7aa30718814cac4f8636 WatchSource:0}: Error finding container 03f38df58414ef8490b0dcb43aac1ecdbc0a538ba63b7aa30718814cac4f8636: Status 404 returned error can't find the container with id 03f38df58414ef8490b0dcb43aac1ecdbc0a538ba63b7aa30718814cac4f8636
Nov 03 02:07:09.884586 ip-10-0-136-112 kubenswrapper[4531]: E1103 02:07:09.884546    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8x65q" containerName="extractor"
Nov 03 02:07:10.243957 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.243921    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ctxbc_11772d42-58e7-41ad-8adb-28220a88c2f9/manager/0.log"
Nov 03 02:07:10.371638 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.371611    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zzws2_2432b11a-588d-42c8-9698-1787ca42ed40/proxy/0.log"
Nov 03 02:07:10.413257 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.413214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x" event={"ID":"66e809e3-4e26-4e74-aa78-9173c5073237","Type":"ContainerStarted","Data":"2765e867c8c2cbe97d07fdc43aa555fd3049543b964346a94042a9790684364f"}
Nov 03 02:07:10.413432 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.413320    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x" event={"ID":"66e809e3-4e26-4e74-aa78-9173c5073237","Type":"ContainerStarted","Data":"03f38df58414ef8490b0dcb43aac1ecdbc0a538ba63b7aa30718814cac4f8636"}
Nov 03 02:07:10.413489 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.413462    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:10.428949 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:10.428892    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x" podStartSLOduration=2.428875715 podStartE2EDuration="2.428875715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 02:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:10.42792493 +0000 UTC m=+5175.641668601" watchObservedRunningTime="2026-11-03 02:07:10.428875715 +0000 UTC m=+5175.642619390"
Nov 03 02:07:11.355324 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.355290    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865gxcxg_65a72581-3f3f-4ff5-b75c-c97eba306705/istio-proxy/0.log"
Nov 03 02:07:11.815180 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.815096    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-46hnd_37b1042d-911e-4e83-b83d-dbb8096646d3/istio-proxy/0.log"
Nov 03 02:07:11.829537 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.829514    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c755ffdd7-8wlk5_0e755136-002b-4519-8ebe-d56ce76b965b/console/0.log"
Nov 03 02:07:11.866877 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.866851    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-d852p_c9342480-831c-4417-8711-a69e79b817a6/istio-proxy/0.log"
Nov 03 02:07:11.903200 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.903153    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-ptjsj_51ac183a-e533-4d5f-91c4-94d05308d76a/istio-proxy/0.log"
Nov 03 02:07:11.995667 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:11.995640    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n_1c6afe96-ef02-4368-b078-6ec4b0e0e1ff/istio-proxy/0.log"
Nov 03 02:07:13.685180 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:13.685010    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcrmw_6a6f16d7-d3db-4415-bf36-c05b4200e860/dns/0.log"
Nov 03 02:07:13.710906 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:13.710879    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dcrmw_6a6f16d7-d3db-4415-bf36-c05b4200e860/kube-rbac-proxy/0.log"
Nov 03 02:07:13.865797 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:13.865759    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wrzvg_2bfad136-a64a-4525-ae3a-d77fb5af1084/dns-node-resolver/0.log"
Nov 03 02:07:14.592846 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:14.592814    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l9wpw_1355bd78-3c21-4bf1-a585-8e66bf03cbb1/node-ca/0.log"
Nov 03 02:07:15.717507 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:15.717474    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865gxcxg_65a72581-3f3f-4ff5-b75c-c97eba306705/istio-proxy/0.log"
Nov 03 02:07:15.975317 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:15.975249    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c9svn_d65cb9f3-5969-41c3-8f50-42bf0f002c72/manager/0.log"
Nov 03 02:07:15.988056 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:15.988021    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/1.log"
Nov 03 02:07:15.988056 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:15.988049    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-sw79f_df529571-d827-4799-9b09-3e0ec8b4ce56/manager/0.log"
Nov 03 02:07:16.037736 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.037690    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-btqdp_20d34f70-7eb9-44e6-a9e4-9ee6d6380d9e/manager/0.log"
Nov 03 02:07:16.140376 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.140339    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-szqjl_dbf70c07-8ae7-4884-933c-17d7dccf6806/manager/0.log"
Nov 03 02:07:16.430635 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.430609    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grd2d/perf-node-gather-daemonset-gk79x"
Nov 03 02:07:16.734317 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.734242    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-46hnd_37b1042d-911e-4e83-b83d-dbb8096646d3/istio-proxy/0.log"
Nov 03 02:07:16.825999 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.825962    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-d852p_c9342480-831c-4417-8711-a69e79b817a6/istio-proxy/0.log"
Nov 03 02:07:16.890149 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:16.890116    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-ptjsj_51ac183a-e533-4d5f-91c4-94d05308d76a/istio-proxy/0.log"
Nov 03 02:07:17.050807 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:17.050722    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n_1c6afe96-ef02-4368-b078-6ec4b0e0e1ff/istio-proxy/0.log"
Nov 03 02:07:20.507972 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:20.507944    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hxbss_43ea7fa2-8908-48c9-a0d5-e10ca4dbc3e7/serve-healthcheck-canary/0.log"
Nov 03 02:07:20.890642 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:20.890610    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865gxcxg_65a72581-3f3f-4ff5-b75c-c97eba306705/istio-proxy/0.log"
Nov 03 02:07:21.213809 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.213730    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x65q_fc1e3ecf-7531-454d-93a5-5a6cce661c27/kube-rbac-proxy/0.log"
Nov 03 02:07:21.238892 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.238864    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x65q_fc1e3ecf-7531-454d-93a5-5a6cce661c27/exporter/0.log"
Nov 03 02:07:21.274572 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.274546    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8x65q_fc1e3ecf-7531-454d-93a5-5a6cce661c27/extractor/0.log"
Nov 03 02:07:21.393696 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.393664    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-46hnd_37b1042d-911e-4e83-b83d-dbb8096646d3/istio-proxy/0.log"
Nov 03 02:07:21.450403 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.450379    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-d852p_c9342480-831c-4417-8711-a69e79b817a6/istio-proxy/0.log"
Nov 03 02:07:21.478220 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.478127    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-th6d7-25rth_89e4978e-91c9-4c21-a10a-aa554ca132ec/insights-gathering/0.log"
Nov 03 02:07:21.488226 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.488146    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-596dc9545f-ptjsj_51ac183a-e533-4d5f-91c4-94d05308d76a/istio-proxy/0.log"
Nov 03 02:07:21.594215 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:21.594180    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cb6869599-sfh7n_1c6afe96-ef02-4368-b078-6ec4b0e0e1ff/istio-proxy/0.log"
Nov 03 02:07:23.874409 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:23.874373    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-9nqbf_a3ce0f76-dc50-4540-9b0f-7daccff99c3d/manager/0.log"
Nov 03 02:07:23.898949 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:23.898926    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7cc9488979-9nqbf_a3ce0f76-dc50-4540-9b0f-7daccff99c3d/copy-manifests/0.log"
Nov 03 02:07:25.289875 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:25.289845    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-s6gsn_ed4ee140-6180-486b-a608-0ff5f28d2355/jobset-operator/0.log"
Nov 03 02:07:25.866480 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:25.866451    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6644c85865gxcxg_65a72581-3f3f-4ff5-b75c-c97eba306705/istio-proxy/0.log"
Nov 03 02:07:26.025386 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:26.025351    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"]
Nov 03 02:07:26.030424 ip-10-0-136-112 kubenswrapper[4531]: I1103 02:07:26.030392    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f979a8c5-kr4cp"]